Dual Metric Contrastive Learning for Self-Supervised Visual Recognition
XXXXXXXXXXXXXXXXXXXX
XXXXXX@XXXX.XXX
Abstract
The Hyperspherical manifold is considered to be apt for acquiring representations contrastively .
The objective function operating on hypersphere has proven its significance from SimCLR [3] with NT-Xent loss.
Still, it’s a question that a single distance function can aid in discriminating the representations on the hypersphere.
The current work is motivated by the question of whether two similarity metrics operating on the same manifold can have significant learning.
We have proposed a new objective function that operates on hypersphere S n with dual metrics.
As we are unaware of the exact representational space of the acquired features, we construct a weighted loss from the Euclidean and Spherical distance metrics. We theoretically prove that these dual metrics can be operated on Hypersphere ( S n ) to learn the visual features.
Although providing a weighted equivalence for the proposed objective function, the performance does not fluctuate with the temperature for disproportionate weighting criterion.
In the second phase, we experiment and attest that DM losses provide significant performance on standard image classification data and are robust to various corruptions and biases.
Finally, this work concludes that dual metric contrastive learning eventually aids the neural networks to have faster convergence rates with all-rounded performance without an increase in computational expense.
1. Introduction
Recently, contrastive self-supervised learning has shown closer performance with supervised approaches.
The underlying reason for competent performance can be due to appropriate augmentations [19, 25, 33], contrastive loss functions [3,30] and the use of memory banks [6,12].
The literature is evident that contrastive representation learning (CRL) has shown its recent success in vast domains [17].
In one of the well-established approaches of contrastive learning, [3], the samples are learned by the deep neural networks with an intuition of increasing the positive pair similarity and decreasing the negative pair similarity simultaneously.
Here, the representations acquired by deep neural networks are operated on the unit hypersphere.
The loss function aids learning by pushing dissimilar representations away from similar ones.
Constricting the final representational space to a unit sphere can provide greater performance for both supervised [28] and unsupervised learning tasks [1].
Finally, maximizing the representational similarity between the positive pairs on the hypersphere significantly affected the learning of the neural networks.
In a contrastive loss function, the similarity distance metric plays a significant role in calculating similarities among the acquired representations.
The spherical distance metrics such as cosine similarity have shown tremendous performance compared to Euclidean, or Manhattan Distances [8,32].
Recent works emphasized the influence of temperature scaling parameters and uniformity of embedding space on performance.
In comparison, this work aims to provide the importance of distance metrics operating on the hyperspherical manifold.
This work proposes a set of new loss functions (Dural Metric losses) for contrastive representation learning by combining two distance functions operating on the hypersphere.
2. Related Works
Hyperspherical Learning
It is well established that, Hyperspherical learning provides generalization in pattern recognition as it constricts the representational space to an n-dimensional sphere [7].
The learning of representations on hyperspherical manifold acquired attention as it leveraged performance by proposing various angular margin objective functions for tackling Face Recognition task [9, 22, 29, 31].
Also, some applications of hyperspherical learning can be seen in few-shot recognition [5,20,21,23].
When uniformly sampled distributions are mapped onto the unit hypersphere, the representations tend to be refined [1].
In variational autoencoders, the latent representations acquired on the Hypersphere are better and more stable compared to that of Euclidean space [8, 32].
Hence, the literature shows hyperspherical learning provides better representations and performance than Euclidean space.
Contrastive Learning Gutmann et al.
[11] has proposed an objective function that learns the distribution of data (in the absence of labels) by discriminating the data distribution with artificially generated noise.
This work had a significant impact and led to develop many contrastive objective (loss) functions that are relatively on par with supervised models.
The contrastive self-supervised learning has provided tremendous throughput with appropriate augmentations [19, 25, 33], contrastive loss functions [3, 30], and the use of memory banks [6, 12].
Some works provided good performance without negative samples [10] and others without the use of projection head [15].
Contrastive Losses Operating on Hypersphere Each component that contributes to the contrastive learning framework is studied extensively.
Here, we constrict to objective functions which operate on the Hypersphere.
We consider these works are closely related to our work.
First, Chen et al.
[3] have provided a standard framework and have utilized an objective function that operates on hypersphere by scaling the radius ( 1 /τ times).
Wang et al.
[30] have proposed two salient properties of contrastive losses, which are alignment and uniformity .
Also, Chen et al.
[4] have factorized the NT-Xent loss [3] into two proportions where one is responsible for the alignment and the other for distribution.
This contrastive loss is assessed with various distribution criteria.
Recently, Wang et al.
[27] have provided some substantial analysis by varying temperature scaling parameters for the NT-Xent loss [3] and clearly detailed tolerance-uniformity dilemma.
3. Contributions
The authors of this work make a distinctive contribution, as mentioned,
This work provides a new family of contrastive loss functions that operate with Dual Metrics (DM).
We theoretically prove that the Euclidean and spherical metrics are equivalent metrics and share the same topological regime on hypersphere .
Empirically proves that, with a slight parametric tuning, DM Losses provide greater performance compared to the standard contrastive loss.
Also, when visualized on S 2 , DM losses tend to distribute uniformly on the hypersphere.
Finally, our experiments prove that DM losses are robust to data biases and corruptions.
4. Method
The section follows by providing an introduction to the contrastive learning framework.
Then, theoretically prove that Euclidean distance metric can be operated on the hyperspherical manifold.
After a theoretical confirmation, we embed the Euclidean distance function into the contrastive loss function and operate the loss function on the hypersphere.
Finally, we provide some perspectives to analyze the significance of each contrastive loss function.
4.1. Contrastive Framework
Contrastive Training Similar to the work by Chen et al.
[3], the contrastive representational framework has similar modules.
As it is clearly evident that augmentations are one of the key contributors to contrastive learning, we augment the data into two views using various augmentation techniques such as Gaussian blur, random resize crop and color jitters, etc.
So, for a given N mini-batch of samples, we generate 2 N samples, and of these, 2 N − 2 samples are considered negative samples.
Hence, we have N − 1 negative pairs to feed the neural network for one positive pair.
The pair of samples (both positive and negative) are fed to the standard neural network (encoder) ResNet50 to acquire the refined representations.
The feature representations acquired by the ResNet50 are vectors in R 2048 space.
The features in R 2048 space are represented as f u i , f
v i and where, i ∈ { 1 , 2 , ..., N } .
As R 2048 space is computationally expressive to operate, the representations are mapped (projected) to a lower dimensional space of Rusing a 2-layered multilayer perception.
These representations are represented as re paired as ( u i , v i ) and where, i ∈ { 1 , 2 , ..., N } .
The pair ( u i , v j ) is said to be positive pair if i = j and else, is said to be a negative pair.
Finally, the mini-batch of features extracted from the two views is now l 2 -normalized and these pairs are represented (˜ u i , ˜ v i ) and where, i ∈ { 1 , 2 , ..., N } .
After normalization, these features are meant to be on a hypersphere of 128 dimensions i.e. S 127 .
Now, these normalized features are contrastively learned using the following objective function,
[Equation]
The loss function in eq (1) is the same as mentioned by
Chen et al.
Where 1 [ i ̸ =
j ] ∈ 0 , 1 is the indicator function which works opposite to that of Kronecker delta i.e.[ i
[Equation]
=
[Equation]
] = 1 if [ i ̸
= j ] and 0 when [ i = j ] .
Linear Evaluation While Linear evaluation, only encoder-learned representations are extracted, and discard the projections.
The encoder weights are frozen and the features f u i , f v i are now attached to 2 Layered MLP for classification.
ℓ
[Figure]
: Euclidian Distance b/w and on Unit Sphere : Spherical Distance b/w and on Sphere of radius : Spherical Distance b/w and on Sphere of radius
SimCLR
Ours
4.2. Dual Distance Metrics for CRL
First, let us consider the well-established spherical and
Euclidean distances.
Let u, v are the vectors in the Euclidean space of d dimension ( u, v ∈ R d ) and the ˜ u, ˜ v
are the unit vectors in d dimensional hypersphere ( S d − 1 ).
[Equation]
As the existing loss L NT − XNT operates on a hypersphere and the similarity metric utilized is cos( d sphere ) .
But, in order to embed the Euclidean distance in the existing loss function, the Euclidean distance metric d euclid and spherical distance d sphere should be topologically equivalent i.e. d euclid , d sphere have to share the metric topology on Hypersphere.
Proposition 1.
The metric topology of S n determined by the Euclidean distance metric d euclid is equivalent to metric topology of S n determined by the spherical distance metric d sphere .
Proof.
Suppose, u, v ∈ S n − 1 and θ ( u, v ) is already mentioned in the equation 4.2.
Where, d sphere ( u, v ) ∈ [0 , π ] .
It can be verified that,
[Equation]
Specifically, θ ( u, v ) is a strictly increasing function of the euclidean distance | u − v | 2 .
It is clear from Theorem 1 (appendix) that θ ( u, v ) is i) non-degenerate ii) non-negative and iii) symmetric .
In order to prove the fourth postulate i.e. triangle inequality let us consider a, b, c ∈ S 2 .
If θ ( a, b ) + θ ( b, c ) ≥ π we obtain , d sphere
[Equation]
a,
[Equation]
a [Equation] ) + d sphere ( b, c ) . Therefore, assume d sphere ( a, b ) + d sphere ( b, c ) ≤ π .
Now consider b as the north pole and rotate the axis- c to c ∗ such that, c ∗ and a are on opposite meridians.
[Equation]
So, the metric space ( S n − 1 , d sphere ) is complete and
[Equation]
From Definition 3 and Proposition 2 1 it can be concluded that,
[Equation]
Hence Proposition 1 implies that Euclidean distance metric and spherical distance metric are equivalent and share the same topological space.
4.3. Dual Metric Losses for CRL
As the euclidean distance and spherical distances both can operate on the same metric topological space (unit hypersphere) we embed these metrics in the contrastive loss function to understand their behavior when operated with
Dual Metrics(DM).
[Equation]
The α and β parameters are weighting functions (hyperparameters) that are to be tuned for optimal loss landscape.
In this work, we evaluate four different settings for α and β parameters and provide our detailed implementations for these choices of parameters which are detailed in Table 2.
Geometric Intuition Now we comprehend the role of similarity metrics in the contrastive loss function geometrically.
For this, let us consider two feature representations acquired from a neural network contrastively as r f 1 , and r f 2 .
These feature representations are l 2normalised ( ˜ r f 1 , and ˜ r f 2 ) and now they lie on unit hypersphere.
Next, cosine similarity is calculated between ˜ r f 1 , and ˜ r f 2 and temperature scaling( τ ) is applied.
The immediate result of τ can be seen as an extension of the radius by a scale of 1 τ i.e. unit hypersphere extends its radius from one to 1 τ .
Now, this temperature-scaled similarity metric is used in the L NT − XNT loss 2 .
But, in this work we also calculate the d euclid for normalised features ˜ r f 1 , and ˜ r fand aggregate them with L NT − XNT loss by appropriate weighting coefficients ( α, β ).
This help to analyze and extract the representations from both the unit-hypersphere and temperature-scaled hypersphere respectively.
An example illustration of our method is visually depicted in Figure.
As we are not aware of the exact embedding space of neural networks but we are trying to map these representations on a unit-sphere with l 2 -norm.
So, our intuition helps to discriminate the representation space by the presence of dual metrics i.e. Euclidean and Spherical.
5. Experiments Loss Parameters
[Equation]
In this section, we run the experiments which outline the empirical performance of DM contrastive losses.
Firstly, we describe all the relevant components and their parameters considered for training the contrastive losses.
Next, we evaluate the performance of DM losses on standard classification datasets and assess whether they vary their performance with temperature alteration.
Also, we assess how samples are distributed on 3-dimensional hyperspherical space ( S 2 ) and observe the relationship between performance and sample distribution across the S 2 .
Lastly, we check the robustness of DM losses when exposed to corruptions and distributional shifts.
5.1. Setup
Data, Network Architecture, and Parameters For evaluating the performance of DM losses we have utilized standard classification data CIFAR-10, CIFAR-100, and TinyImageNet (ImageNet-200).
In the contrastive framework, the augmentations, encoder and projection head, and the hyperparameters for SimCLR and DM losses are kept identical for a fair evaluation.
For contrastive training, the augmentations such as random resize crop, random horizontal flip, random grayscale, and color jitter are applied.
The
ResNet-50 is used as the encoder for the base encoder and for the projection head, 2-Layered MLP with 2048 toneurons is utilized.
LARS optimizer is applied with a learning rate of 0 .
3 × batch 256 applied.
For CIFAR-10 anddatasets, 1028 samples are trained as a batch.
For training ImageNet-200 contrastively we’ve used a batch size of 256.
Also, we’ve applied linear warmup for the initialepochs and used a cosine scheduler for decaying learning rates without any restarts.
If not mentioned specifically, we have used the temperature scaling of τ = 0 .
If not mentioned particularly, the models are trained contrastively for 120 epochs.
After contrastive training, we perform the linear evaluation and for this, we consider augmentations such as random resize crop, random horizontal flip, and normalization.
Then freeze the trained encoder which is trained contrastively and attach a 2-Layered MLP (2048 to the number of classes) for classifying the representations using softmax activation.
A dropout is used between 2-Layered MLP with a drop rate of 40%.
For the linear evaluation, SGD is used to optimize the network with a momentum of 0.9 and with a learning rate of 0 .
1 × batch 256 .
The learning rate is sched- [Table] uled at multiple steps i for every by scale of 0.1.
If not mentioned particularly, the models undergo linear evaluation for 90 epochs.
5.2. Performance
As mentioned, to evaluate the performance of DM losses we have utilized standard classification data CIFAR-10,
CIFAR-100, and ImageNet-200.
From the results illustrated [Table] in Table 3, one can infer that without any additional increment in computational expense the DM losses perform superior for most of the scenarios.
Specifically, the third version of DM loss L DM 3 has shown greater performance in most of the scenarios.
5.3. Temperature Alteration
But, these experiments are performed under optimal temperatures ( τ = 0 . 07 ).
It is necessary to assess the performance for a broad spectrum of temperatures to judge the model’s stability.
Thus, we fluctuate the temperature τ from 0.01 to 1 and observe the stability of the DM losses.
From the results demonstrated in Table 1, it can be understood that even with varying temperatures most of the DM losses have stable learning and again L DM 3 has provided significant performance for both CIFAR-10 and CIFAR-100 datasets respectively.
Thus with these evaluations, we infer that DM losses have learning stability even with altering temperatures.
5.4. Hyperspherical Distribution
It should be noted that L DM 1 has comparatively poor performance and does not illustrate the surge of learning.
The reason for the superior performance of L DM 3 and substandard performance of L DM 1 can be comprehended by visualizing the sample distribution on the hypersphere.
To understand the behavior of representations, we provide two different visualizations
• Hyperspherical Diffusion (Sample distribution on S 2 ). • Manifold Visualisation using UMAP 3 .
Hyperspherical Diffusion
In the first visualization, the data samples which are fed into a neural network are mapped onto S 2 depicts the visual representations that are distributed on unit 2-sphere i.e. S 2 .
To visualize these representations on S 2 , the contrastive framework is modified accordingly for the CIFAR-10 dataset 4 .
First, we use ResNet18 as the base encoder and used 3Layered MLP ( 512 → 64 → 3 ).
Hence, while training contrastively we get a pair of feature representations at the terminal layer in 3 dimensions R 3 (Refer Figure ?? .
After l 2 − normalisation the feature vectors occupy the S 2 space.
These feature vectors are directly visualized after training the neural network contrastively for 40, 80, and 120 epochs respectively.
Now, in Figure 4 we visualize the test samples which are unseen by the model.
One can observe that each of the samples diffuses on S 2 i.e., the samples have the tendency to occupy the S 2 .
This gives a vivid picture of DM losses and illustrates that they tend to diffuse across the S 2 space with the right choice of the parameters α, and β .
Specifically, the L DM 3 have a high tendency to spread uniformly across S 2 with increasing epochs.
Rather than just relying on visual facets illustrated in Figure 4, we examine the performance of each loss function for all the mentioned epochs.
Uniformly distributed samples on S 2 seek to have greater performance. The linear evaluation accuracy scores are detailed in Table 4 justifies that samples that diffuse over the hypersphere perform better.
This is obliged by integrating a good loss function that provides well-discriminative decision boundaries. As L DM 3 improves the uniformity of distributed samples over [Figure]
Loss [Table] the hypersphere with well-discriminative decision boundaries leading to better empirical performance.
5.5. Robustness
The Neural Networks are tested for their robustness for safety-critical applications; thus, we assess whether the proposed models are robust.
In this work, we assess our models by considering two types of robustness, i.e., Corruptions,
Distributional Shifts, and Data Biases.
Corruptions To evaluate the robustness to corruptions, we consider the ImageNet-C dataset [13].
This data has four major categories of corruptions (Noise, Blur, Weather, and Digital), and each category is again divided into subcategories.
Also, each sub-category has five severity levels from 1 to 5 (1 resembles the minimum, and 5 is the maximum severity).
Now, we consider the SimCLR model as the baseline model and evaluate our losses accordingly.
Also, we evaluate with two augmentations.
The first augmentations are the same as the previous, and the second is AugMix [14].
These augmentations are performed during linear evaluation, and the encoder weights are frozen (The encoder is trained on
ImageNet-200).
The results are evaluated on the two metrics: mean Corruption Error (mCE) and relative mean Corruption Error (rel. mCE).
From results illustrated in Table 5, with standard augmentations L DM 3 has less error compared to SimCLR.
Also, when AugMix is used, L DM 3 has very low mCE, but almost all the losses have similar rel.
mCE 5 .
Cumulating these results, we say L DM 3 is robust to corruptions.
when trained on self-supervised contrastive losses is one of the challenges to providing a safe AI [26].
Hence to assess the robustness of models to biases we consider two synthetic datasets Colored MNIST, Corrupted CIFAR and one real-world dataset– Biased FFHQ [18].
The diversity ratio in each of these datasets ranges from 0.5% to 5% (Except for Biased FFHQ).
Increasing diversity among the samples [Table] has proven significant performance and eventually provided better de-biased representations.
As self-supervised contrastive learning does not rely on labels, it is crucial to understand the representations acquired from biased data (A comprehensive evaluation of various alignment and conflict samples are detailed in the appendix).
So from Table 6 it can be seen that DM contrastive loss outperforms every biased dataset.
Also,
L DM 3 has provided significant performance in most of the scenarios.
Also with our analysis, we say that DM losses provide better performance with conflicting samples.
From these results, neural networks trained on DM contrastive losses provide incremental robustness to Corruptions and Data Biases.
Hence we justify that, adding an additional euclidean distance metric (operating on S d − 1 ) can provide finer performance not just on standard image recognition, but also enhance the robustness of the model.
6. Discussion
Gutmann et al.
[11] has proposed an objective function that learns the distribution of data (in the absence of labels) by discriminating the data distribution with artificially generated noise.
This work motivated to develop many objective (loss) functions that are relatively on par with supervised models.
The self-supervised contrastive learning has been viewed from many perspectives, and each of these perspectives has an intuitive conception to understand the representations.
Wang et al.
[30] has proposed two properties of contrastive loss, which are alignment and uniformity .
Likewise, considering DM losses, they are embedded with alignment parameter, i.e., alignment is obtained by factorizing the equation (4). [Equation]
(cid:124) (cid:123)(cid:122) (cid:125) weighted alignment +
[Equation]
log   2 N (cid:88)
j = 1 [ i ̸
=
j ] e ˜ u
Ti ˜ v j /τ 
 +
[Equation]
log   2 N (cid:88)
j = 1 [
[Equation]
|
[Equation]   Specifically, in equation (5), it is clear that DM losses justify the alignment property of contrastive losses.
Specifically, the term weighted alignment has distinct parameter values i.e., α, β .
Although the weight α is operating on spherical distance (i.e. - α (˜ u Ti ˜ v i /τ ) ), according to Lemma 1 those distances satisfy isometry in S d − 1 .
Hence, the weighted alignment property is a key contributor to the performance.
Now, other parts of the loss function, other than the weighted alignment, can be closely related to diffusion property.
This property can be closely considered as distribution property [4].
Hence, we believe that hyperspherical diffusion can be a decisive component to not just provide better performance but also better robustness.
As we do not attempt to guarantee the theoretical formulations [20] but rather envision an intuition from the vivid visualizations.
Hence, the work by Chen et al.
, [4] is also considered to be closely related to our work.
The temperature scaling parameter, τ is another important factor in the contrastive loss that proved to have tremendous performance [3].
This τ parameter should be chosen appropriately to determine the optimal performance.
But,
Wang et al., [27] conducted a study that details the importance of temperature parameters.
First, by providing the theoretical analysis at both extremities τ (i.e. τ ∈ (0 , + ∞ ) .
Second, the authors have experimentally proved that an effective temperature parameter would be τ = 0 .
Also, they contributed a tolerance property which gives an intuition to choose the apt temperature.
Hence we conduct experiments with this optimal temperature of 0 . 3 and assess whether the proposed loss can still sustain random temperature fluctuations.
Now, Table 7 compares the above-mentioned loss functions with the DM losses.
All the evaluations are fairly evaluated without any alterations in the temperature parameter ( τ ), augmentations and also other influencing hyperparameters.
The parameters related to the loss functions are considered and evaluated at their optimal setting except for the parameters that are chosen optimal as per authors’ claims.
For instance, Wang et al.
[27] has illustrated clearly that, the loss function performs optimally at τ = 0 .
and we considered it accordingly Also, the weights of L align and L uniform are chosen according to their optimal performance.
[Table]
So in most cases, L DM 3 competes and provides significant performance.
From the results obtained from Tables 3, 1b, 4, 5, 6, 7 the L DM 3 loss do not compromise on performance and robustness at any level.
Also, the weighting parameters α, β are notable components as they determine the success rate of DM losses.
When there is a higher weight for the loss function, which operates using spherical distance (eg.
L DM 1 ) then the performance fluctuates, and the samples do not quickly diffuse onto the hypersphere to distribute themselves.
Also, when there are non-homogeneous weights ( α + β ̸ = 1 and α, β ≥ 1 ) the DM losses tend to perform poorly 6 .
Hence, L DM 3 is an apt contrastive loss for most of the downstream self-supervised tasks. [Table]
7. Conclusion
These DM contrastive losses were interpreted from their underlying geometrical significance but, have shown their leveraging performance on standard image classification data, biased data and data with corruptions.
The key contribution of DM losses is that they do not need any additional computational resource and provides better performance under various scenarios.
The experiments prove that
DM losses do not fluctuate in their performance with altering τ and they provide a significant performance of standard classification and robustness tasks.
It should be noted that
DM losses are sensitive to α, β as they control the distance metrics operating on the hypersphere.
To have significant performance, a lower weight is given for loss operating on spherical distance and a higher weight for the loss operating with Euclidean distance ( L DM 3 ).
Appendix Broader Impact
A simple geometric distance function can enhance the performance for the considered downstream tasks.
The scope for DM losses is also into upstream tasks applications but not limited to image denoising, segmentation, reconstruction and generation.
Our method does not extensively increase computational resources to excel in the performance but, provides a simple geometric trick and improves baselines.
Also, the authors have firmly decided to contribute to safe AI and thus strive to reduce AI biases.
The self-supervised learning methodically relies on the intrinsic characteristics of the given data.
Hence, to provide safe AI to the community, the authors ensure that the model is robust to some of the safety-critical aspects.
A stronger motivation arises when Deep Learning continues to be applied in various technologies and social domains.
7.1. Theoretical Proofs and Prerequisites
Some of the fundamental definitions, of theorems, are adapted from the relevant sources.
To have a fair understanding of metric spaces refer [2] and to have a topological perspective of the metric spaces refer [16].
The metrical properties of Euclidean and spherical manifolds can be extracted from the work by Ratcliffe et al.
The reader can follow the appendix progressively as sufficient fundamentals for the current work are detailed precisely.
Definition 1.
Suppose a non-empty set X / ∈ ϕ and for each x 1 , x 2 ∈ X let d ( x 1 , x 2 ) be a real number,
Non-degenerate d ( x 1 , x 2 ) =
iff x 1 = x 2 ; 2.
Non-negative d ( x 1 , x 2 ) ≥ 0 ; 3.
Symmetric d ( x 1 , x 2 )
=
d ( x 2 , x 1 ) ∀ x 1 , x 2 ∈ X. 4.
Triangle Inequality d ( x 1 , x 3 ) [Equation] d ( x 1 , x 2 ) + d ( x 1 , x 2 ) ∀ x 1 , x 2 , x 3 ∈ X .
Then ’ d ’ is said to be a metric (distance) on space X and ( X, d ) is called a metric space.
Definition 2 Suppose ( X, d ) is called a metric space and let T d be the collection of subsets of U of X such that, for each x ∈ U ∃ r > 0 with a open ball B ( x ; r ) ⊂ U .
Then ( X, T d ) is called the topological space defined under the metric (distance) d .
Definition 3 Suppose d A , d B be distance metrics on X with topologies T A , T B respectively .
Then d A and d B metrics are equivalent iff they have the same topology i.e.
T A ≡ T B .
Proposition 2 Let the distance metrics d A , d B on X are such that for some ϵ we have,
[Equation]
[Equation]
Where, ∀ x 1 , x 2 ∈ X .
Then these metrics d A , d B are equivalent metrics .
Proof.
Let T A , T B be the topologies defined by metrics d A , d B respectively.
We must show that a subset of U of
X ∈ T
A iff it ∈ T B .
Let, U ∈ T A and u ∈ U .
There exist some r 1 > 0 such that B d A ( u ; r 1 ) ⊂
U i.e.
{ u ∈ X | d A ( u, v ) <
r 1 } ⊂ U.
Similarly consider B d B ( u ; r 2 ) where r 2 = r 1 /ϵ .
If v ∈
B d B ( u ; r 1 /ϵ )
then d B ( u, v ) <
r 1 /ϵ .
But, 1 ϵ d A ( u, v ) ≤ d B ( u, v ) and so, for v ∈ B d B ( u ; r 1 /ϵ )
we have d A ( u, v ) ≤ d B ( u, v ) ≤ ϵ × r 1 ϵ
=
r
Hence, v ∈ B d A ( u ; r 1 ) whenever v ∈ B d B ( u ; r 1 /ϵ ) but,
B d A ( u ; r 1 ) ⊂
U and so, B d B ( u ; r 1 /ϵ )
⊂
B d A ( u ; r 1 ) ⊂
U
Thus, for u ∈ U , there exist some r 2 > 0 ( r 2 = r 1 /ϵ such that, B d B ( u ; r 2 ) ⊂ U .
Thus, U is open in the topology determined by metric d B i.e. U ∈ T A and U ∈ T B .
As
U ∈ T B for uinU ∃ r 1 > 0 with B d B ( u ; r 1 ) ⊂
U .
If d A ( u, v ) <
r 1 /ϵ we have d B ( u, v ) ≤
ϵd A ( u, v ) <
ϵ × r 1 ϵ .
So, B d A ( u ; r 1 /ϵ )
⊂ B d B ( u ; r 1 ) ⊂ U .
Thus, U ∈ T A iff U ∈ T B and ∴ T A ≡ T B
Definition
The spherical distance function d sphere is a metric on hypersphere of d dimensions ( S d − 1 ).
Proof.
Let u, v are the vectors in the Euclidean space of d dimension ( u, v ∈ R d ) and the ˜ u, ˜ v
are the unit vectors in d dimensional hypersphere ( S d − 1 ).
The spherical distance d sphere is written as,
[Equation]
sphere
[Equation]
[Equation]
u T
[Equation]
v )
The spherical metric d sphere is non-negative, nondegenerate, and also symmetric.
Now we prove the triangle inequality to justify that, ( S d − 1 , d sphere ) forms a metric space.
It should be noted that the orthogonal transformations of R d → S d − 1 preserves spherical distances.
So, we transform u, v, w by an orthogonal transformation and u, v, w ∈ R d .
Here to prove this inequality let us consider d = 3 then we have,
[Equation]
[Equation]
Thus we obtain θ ( u, w ) ≤ θ ( u, v ) + θ ( v, w ) .
Lemma 1 A function f : S d − 1 → S d − 1 is an isometry iff it is an isometric w.r.t d euclid on S d − 1 because | u − v | 22 ≡ 2(1 − u.v ) .
Proof.
As, u, v ∈ S d − 1 it should be clear that, | u | = | v | = 1 .
[Equation]
So, | u
[Equation]
Gradient Analysis of losses
The gradients are calculated and analyzed w.r.t positive pairs to comprehend the target distribution similar to Chen et al.
First, we calculate the gradients w.r.t +ve samples for L NT − Xent and then we’ll further proceed for DM loss
L DM .
Gradients w.r.t +ve samples for L NT − Xent
L NT [Equation] Xent
[Equation] − log   (cid:88) v ∈{ [Equation] − } exp(
[Equation]

 ∂
∂u ( L
[Equation]
Xent ) =
[Equation]
∂u
[Equation]
) − ∂ ∂u   log (cid:88) v ∈{ v + ,v − } exp( u T
[Equation]
[Equation]

[Equation]
Suppose, Z ( u, v )
[Equation]
(cid:88)
[Equation]
τ
Here, Z ( u, v ) can be assumed as the partition function [11] for the contrastive loss.
[Equation] t +ve samples for L DM
[Equation]
[Equation]