O N
THE INADEQUACY OF OPTIMIZING ALIGNMENT AND UNIFORMITY IN CONTRASTIVE LEARNING OF SENTENCE REPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive learning is widely used in areas such as visual representation learning (VRL) and sentence representation learning (SRL).
Considering the differences between VRL and SRL in terms of negative sample size and evaluation focus, we believe that the solid findings obtained in VRL may not be entirely carried over to
SRL.
In this work, we consider the suitability of the decoupled form of contrastive loss, i e., alignment and uniformity , in SRL.
We find a performance gap between sentence representations obtained by jointly optimizing alignment and uniformity on the STS task and those obtained using contrastive loss.
Further, we find that the joint optimization of alignment and uniformity during training is prone to overfitting, which does not occur on the contrastive loss.
Analyzing them based on the variation of the gradient norms, we find that there is a property of “gradient dissipation” in contrastive loss and believe that it is the key to preventing overfitting.
We simulate similar “gradient dissipation” of contrastive loss on four optimization objectives of two forms, and achieve the same or even better performance than contrastive loss on the STS tasks, confirming our hypothesis.
The codes and checkpoints will be released for study upon acceptance.
I NTRODUCTION
Unsupervised contrastive learning Wu et al.
(2018) is originated from visual representation learning (VRL)
Chen et al.
; He et al.
(2020); Grill et al.
and has achieved impressive performances therein.
Briefly, contrastive learning forces the representation of an input instance (or “anchor”) to be similar to that of an augmented view of the same instance (or “postive example”) and to differ from that of a different instance (or “negative example”).
One plausible justification of this approach is that minimizing the loss of contrastive learning (or “contrastive loss”) is shown to be equivalent to simultaneously minimizing an “alignment loss” and a “uniformity loss”, where the former dictates the representation similarity between an instance and its positive examples, and the latter forces the representations of all instances to spread uniformly on the unit sphere in the representation space Wang & Isola (2020).
Notably this decomposition of the contrastive loss relies on the condition that the number of negative examples participating in the contrastive loss approaches infinity.
For VRL, it is arguable that such a condition holds reasonably well since usually a large number (e.g., 65536 He et al.
of negative examples are used in a batch.
In recent years, contrastive learning has also been adapted to sentence representation learning (SRL), by fine-tuning the representations obtained from a pretrained language model (e.g. BERT Devlin et al.
Yan et al.
; Giorgi et al.
; Gao et al.
; Zhang et al.
(2022b).
Such approaches have demonstrated great performances not only for downstream classification tasks, but also for semantic textual similarity (STS) tasks.
It is noteworthy that significantly contrasting VRL, which are primarily evaluated using downstream classification tasks Russakovsky et al.
Krizhevsky et al.
(2009), or via “extrinsic” protocols Chiu et al.
; Faruqui et al.
SRL particularly emphasizes the STS tasks, or “intrinsic” protocols, when evaluating the quality of learned sentence representations Reimers & Gurevych (2019);
Li et al.
Yan et al.
Zhang et al. (2022c).
This is because the representations obtained from the pretrained language models have already shown strong transfer capability to downstream tasks Devlin et al.
Liu et al.
(2019) while their similarities are rather poorly correlated with the human-rated similarities
Reimers & Gurevych (2019); Li et al.
Following the justification of contrastive learning in VRLWang & Isola (2020), some works Gao et al.
; Zhang et al.
(2022b) attribute the success of contrastive learning on the STS tasks to a good balance between alignment and uniformity .
Consequently, alignment and uniformity losses are adopted widely as the key metrics for evaluating the goodness of sentence representations learned from contrastive learning Gao et al.
; Zhang et al.
(2022b;c;a)
; Klein & Nabi (2022).
Noting that contrastive learning for SRL in fact only uses a rather small number of negative examples (e.g, 63 Gao et al.
; Zhang et al.
(2022b) or smaller Zhang et al.
(2022c))
, in this paper, we question whether the “decomposition principle”, or jointly optimizing alignment and uniformity , adequately explains the performance gain in the STS tasks brought by contrastive learning.
After extensive experiments, we find that optimization using alignment and uniformity losses produces lower performance than that with contrastive loss in the STS tasks.
Moreover, this performance degradation is not reflected by the alignment and uniformity metrics.
Interestingly, we also observe the same phenomenon in contrastive learning with another loss function, Decoupled Contrastive Loss (DCL)
Yeh et al.
(2021), in which the optimization objectives are very similar to alignment and uniformity .
Our further studies also show that training with such decoupled forms of contrastive loss cause severe overfitting, which does not occur in training with the standard contrastive loss.
These observations suggest that alignment and uniformity losses might not serve suitable substitutes for contrastive loss in SRL and that the success of the standard contrastive learning for SRL can not be adequately explained in terms of alignment and uniformity properties or some delicate balance between the two.
This paper focuses on uncovering other important factors, beyond alignment and uniformity , that contribute to the success story of contrastive learning in SRL and explain the performance gap between the training scheme using the contrastive loss and those using a decoupled contrastive loss (in terms of alignment and uniformity or their equivalent).
Specifically, we hypothesize that the training dynamics of the standard contrastive learning for SRL plays an essential role in its effectiveness.
To that end, we decompose the gradient of the contrastive loss into an alignment component and a uniformity component and compare the norms of these two components with their counter-parts in training with the decoupled contrastive losses.
Interestingly, we observe a distinct “gradient dissipation” phenomenon in training with the standard contrastive loss: the gradient signal quickly drops and vanishes as soon as the negative example is adequately further away from the anchor than the positive example, where the adequacy appears to be reflected by a rather moderate threshold.
Notably such a phenomenon does not appear in the training schemes using a decoupled contrastive loss, when the negative sample size is small.
This observation led us to believe that “gradient dissipation” plays an essential role in standard contrastive learning.
To validate this hypothesis, we construct two new loss functions, both capable of inducing “gradient dissipation” in their training dynamics.
We test them experimentally and observe that indeed training with both losses gives comparable or even better performance in the STS tasks than the standard contrastive loss.
Interestingly, similar to the contrastive loss, training with these new loss functions also eliminates the alleviates the overfitting problem observed in training using decouple contrastive losses.
This confirms that gradient dissipation serves a key role in the success of contrastive learning for SRL and suggests that properly conditioning the training dynamics is more important than optimizing alignment and uniformity when not too many negative examples are used.
Along our development, we provide insights as to why gradient dissipation is a desirable property.
We also provide additional theoretical justifications on the effectiveness of the new loss functions by showing that they are in fact upper bounds of the standard contrastive loss.
Due to page limit, some results, derivations and discussions are presented in Appendix.
P RELIMINARY 2. C ONTRASTIVE L EARNING IN S ENTENCE R
L EARNING
Let C be a set of sentences.
For each x i ∈ C , we use x ′ i to denote an augmented view of x i .
With respect to anchor x i , x ′ i is referred to a positive example, and any x j or x ′ j ( j ̸ = i ) is referred to as
[Table]
a negative example.
There is an encoder mapping each x i and x ′ i to their representations
h i and h ′ i , which are vectors in R d .
Considering the metric invariance, we usually normalize them to obtain ˆ h
i and ˆ h
′ i , which are constrained on the unit hypersphere S d − 1 centered at the origin.
InfoNCE
Loss Oord et al.
(2018) or “contrastive loss” for anchor x
i is defined by
[Equation]
where τ is the temperature hyperparameter and N represents the number of negative samples.
The quality of sentence representations is shown to insensitive to N Gao et al.
(2021), and a small N , such as 63 Gao et al.
; Zhang et al.
(2022b) or smaller Zhang et al.
(2022c) has shown to be the same sufficient as the bigger one.
2. T HE D ECOUPLE V ERSION OF C ONTRASTIVE L OSS
Alignment and uniformity Wang & Isola (2020) decoupled from contrastive loss are shown to be two significant properties.
The losses based on the two properties are defined as
[Equation]
where α , t and λ are three hyperparameters.
In the meantime, we follow another work Yeh et al.
(2021) on decoupled contrastive learning, which removes the positive sample part from the denominator of contrastive loss:
[Equation]
Note that the above two decoupled forms of contrastive loss are treated equally in this paper, since the alignment part are recognized as equivalent Yeh et al. and we proved the uniformity part of them have the same lower bound 1 , which corresponds to the optimization objective of Minimum
Energy Problem on the hypersphere Kuijlaars & Saff (1998); Liu et al.
[Figure] [Figure]
All above images are plotted based on the optimization process for BERT base .
2. P ERFORMANCE C OMPARISON
We compare the performance of contrastive loss and its two decoupled forms based on SimCSE Gao et al.
(2021), and evaluate with seven datasets on semantic textual similarity (STS) tasks and seven downstream classification datasets on transfer (TR) tasks from the SentEval toolkit Conneau & Kiela (2018).
We report the average Spearman correlation for the STS tasks and the average accuracy for the TR tasks in Table 1 .
Please refer to Appendix A for experimental details.
As can be seen from the results, we can find that (1) the performance on both STS and TR tasks is better than original pretrained models after optimizing with any one of the three loss functions; (2) the main improvement over the original pretrained models is on the STS tasks, while the improvement on the TR tasks is relatively weak; (3) the decoupled forms obtain the same or better performance on TR tasks as contrastive loss, but there still have a performance gap between them.
These observations reflect to some extent the validity of alignment and uniformity to replace contrastive loss on TR tasks, but it also implies that there are some other factors playing a key role in the improvement of STS tasks, which have been neglected in the previous studies.
T HE INADEQUACY OF OPTIMIZING ALIGNMENT AND UNIFORMITY 3. P ROBLEM A NALYSIS
To gain a deeper understanding for the above performance gap, we record several metrics during the training and evaluation process, the images of which are shown in Figure 1.
Figure 1a shows the L uniform - L align scatterplot of sentence representations on STS-B Cer et al.
(2017) development set, where the colors represent the average Spearman correlation of the STS tasks.
It is important to note that the scatterplots of this style are widely used to show that the proposed method is able to achieve a better balance between alignment and uniformity than other methods Gao et al.
; Zhang et al.
(2022a;b;c);
Klein & Nabi (2022).
Indeed, the scatterplot shows that the sentence representations whose ( L uniform , L align ) located in the middle of the image perform better than those whose location in the top left or bottom right.
However, the performance gap between different loss functions are not reflected by the locations in this figure.
Specifically, the sentence representations pretrained by L cl and its decoupled forms have the almost same alignment and uniformity , but different performance on the STS tasks.
Figure 1b shows the numerical variation of L cl , L align and L uniform on the training set and development set during the optimization process with L cl .
In terms of trends, all metrics on both the training set and development set first decrease and then remain flat, which can be seen as a reference for normal training in the study.
Figure 1c and 1d show the same metrics during the optimization process with L a&u and L dcl .
Comparing with Figure 1b, the variation trends of the two figures show different degrees of overfitting.
Specifically, when L a&u is applied for optimization, all metrics except L align show first decrease and then increase in the development set, while the similar situation occurs in all three metrics when
L dcl is applied.
3. O PTIMIZATION D YNAMICS S TUDY
Since L a&u and L dcl can be treated as equivalent to L cl only when negative samples tend to infinity, we need to pay attention to what exactly is the difference between them when the number of negative samples is small.
To gain more insights on their differences, we investigate the gradient property between contrastive loss and its decoupled forms in optimization dynamics.
The gradient of L cl for the anchor h i can be split into ∇ poscl and ∇ negcl :
[Figure]
[Table]
(6) where I is the identity matrix and M h
i is the projection matrix on h i .
The first term ∇ poscl points from ˆ h i to ˆ h
′ i , “pulling” the anchor and the positive sample close to optimize alignment , while the second term ∇ negcl points from ˆ h i to ˆ h ′ j , pushing the anchor away from the negative samples to optimize uniformity .
Then the gradient norm on alignment is computed:
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Equation]
ij
[Equation]
[Equation]
[Equation]
[Equation]
where θ ii ′ represents the angle between the anchor ˆ h i and the positive sample
ˆ h
′ i , and θ ij ′ represents the angle between the anchor ˆ h i and the negative sample
ˆ h ′
j .
Likewise, we give the gradient norm of L align and L dcl on alignment :
[Equation]
Comparing equation 7, 8 and 9, we find that the gradient directions of these three parts are the same, but a key difference among them is that θ ij ′ is contained in ∥∇ poscl ∥ but not in ∥∇L align ∥ neither in ∥∇ posdcl ∥ , which will cause the gradient signal on alignment to be independent of θ ij ′ .
Figure 2a visualize this difference by plotting the heatmaps of ∥∇ poscl ∥ , ∥∇L align ∥ and ∥∇ posdcl ∥ with θ ii ′ - θ ij ′ .
The leftmost four plots show the ∥∇ poscl ∥ under the different number of negative samples, while the rightmost two plots show ∥∇L align ∥ and ∥∇ posdcl ∥ under the condition ofnegative sample size.
Observing the leftmost four plots, we can find that the area of the shaded part (or the area with weak gradient signals) gradually decreases as the number of negative samples increases.
Therefore, it is conceivable that when the number of negative samples tends to infinity, these images will gradually become identical to the two rightmost images.
[Figure]
|| poscl
||
(N
[Figure]
ii|| poscl ||
(N
[Figure] [Figure]
ii|| poscl ||
(N=
[Figure]
) ij ii|| poscl ||
(N
[Figure] [Figure]
[Figure]
ii||
[Figure]
|| (N
[Figure]
|| posdcl || (N
[Figure]
[Figure] [Figure]
respect to the alignment part.
When the images are plotted, the 1 / ∥ h i ∥ in equations are ignored; all θ ij ′ in one equation are treated as the same value; all hyperparameters in equations are consistent with the columns corresponding to BERT base in Table 2.
Comparing the leftmost plot with the rightmost two plots, we can clearly see the difference between them, where the former has a “gradient dissipation” situation that the latter does not have.
So let us think what will happen during training.
θ ij ′ will increase and θ ii ′ will decrease gradually.
For ∥∇ poscl ∥ , the gradient signs of some anchors will suddenly dissipate at a certain time and only the anchors whose θ ij ′ is not too larger than θ ii ′ will still receive the gradient signals.
For ∥∇L align ∥ and ∥∇ posdcl ∥ , since they are not constrained by θ ij ′ , their gradient signals will do not vary with the horizontal axis.
All anchors except those with absolutely small θ ii ′ will receive a continuous gradient signal during the training process.
We point out that the above qualitative conclusion does not fail to hold if the angle is replaced with a generic distance function ρ ( ., . )
on the hypersphere.
Therefore, the difference on “gradient dissipation” can be intuitively described as: “When the number of negative samples is small, contrastive loss tries to keep ρ ( ˆ h i , ˆ h
′ i ) small relative to ρ ( ˆ h
i , ˆ h ′ j ) by some value, while its decoupled forms try to keep θ ii ′ absolutely small”.
Here, we unthinkingly hypothesize that the “gradient dissipation” is a key property in the performance gap between contrastive loss and its decoupled form, and more insights on this hypothesis will be provided in Section 5.
E XPERIMENTAL VERIFICATION 4. V ALIDATION VIA THE INTRODUCTION OF L dcl +
To validate our hypothesis, we try to simulate this property with the smallest possible changes.
Then we notice two facts: (1) L dcl will gradually become negative as training proceeds; (2) the “gradient dissipation” only occurs when θ ij ′ is large enough with respect to θ ii ′ .
These two facts inspire us to introduce a ReLU function Glorot et al.
(2011) on top of L dcl to force the truncation of the gradient signal of the anchor when L dcl < 0 :
[Equation]
Likewise, we can obtain its gradient norm associated with alignment :
[Equation]
The leftmost plot of Figure 2b shows (cid:13)(cid:13) ∇ posdcl + (cid:13)(cid:13) under the condition of 64 negative sample size.
Comparing with the leftmost plot of Figure 2a, we can the images of (cid:13)(cid:13) ∇ pos dcl + (cid:13)(cid:13) and ∥∇ poscl ∥ are almost identical, which proves that the added ReLU function can indeed simulate the property of “gradient dissipation”.
More importantly, the experimental results in Table 1 demonstrate the effectiveness of
L dcl + , which boosts 3%-7% on the STS tasks in the different pretrained models.
4. V ALIDATION VIA THE UPPER BOUNDS FOR L cl
In fact, L dcl + is introduced not only for the above mentioned observations, but also because it can be proved to be the upper bound of L cl .
Further, we find that another upper bound with the property of “gradient dissipation”, which can be derived after a second relaxation based on L dcl + :
[Equation]
where τ log( N − 1) can be replace as the margin hyperparameter m .
Then we can get a new loss function:
[Equation] [Equation]
Note that the new upper bound is very close in form to Tripet Loss Weinberger & Saul (2009), but the most hard negative sample is selected for optimization in this form, so we call the loss function
L mpt (Minimum dot Product Triplet Loss).
To increase the variation diversity of gradient norm, we replace the dot product with the Euclidean distance to obtain L met (Minimum Euclidean distance Triplet Loss) and the angle to obtain L mat (Minimum Angle Triplet Loss):
[Equation]
We also report their results in Table 1.
We find that the performance of these loss functions ahead of the decoupled forms of contrastive loss, and is comparable to or better than contrastive loss.
Likewise, their gradient norm associated with alignment , (cid:13)(cid:13) ∇ posmpt (cid:13)(cid:13) , ∥∇ posmet ∥ and ∥∇ posmat ∥ , are plotted in
Figure 4c.
There are some difference among these gradient norms due to the choice of different distance functions, but it does not bring about a significant difference in their performance.
Therefore, the effectiveness of the “gradient dissipation” property is further affirmed by controlling variables.
4. M ECHANISTIC ANALYSIS OF GRADIENT DISSIPATION
We develop a interest in L mpt because the degree of “gradient dissipation” can be adjusted by the parameter m (Figure 2b) and independent of the number of negative samples.
Therefore, we can quantitatively observe how the value of margin effect the performance with this more flexible form.
Recalling that the “gradient dissipation” occurs when d ij = ∥ ρ ( ˆ h i , ˆ h ′ j ) − ρ ( ˆ h i , ˆ h ′ i ) ∥ is outside a certain range, in L mpt , where ρ ( ., . ) is a distance function.
Therefore, we can obtain different mean value of d ij by selecting different m during training. Figure 3a shows that how the performance on STS tasks and TR tasks varies with d ij . Then we can find that the performance of STS tasks are much more sensitive to d ij than the TR task’s, and too large or too small a d ij will cause the performance of the STS task to drop sharply.
In contrast, the performance of the TR task does not vary significantly with d ij , which seems to explain to some extent why the decoupled form can work well in VRL. What’s more, it can be seen that a suitable margin value helps to achieve a doublebest performance for both STS and TR tasks, which indicates that sentence representation quality can achieve some sense of synthetic optimality and that there is no significant mutual exclusivity between intrinsically and extrinsically evaluated performance.
To better understand the first observation, we set m to 0.10, 0.23 (corresponding the best performance) and 0.80 respectively, and record the numerical variation of L cl , L align and L uniform on the training and development sets.
Comparing Figure 3c and 3d first, which reflect the cases when m is 0.10 and 0.23.
We find that L cl can drop to lower on both training and validation sets when m is 0.23, and this difference is magnified by the L uniform , while L align remains almost constant.
It is consistent with our intuition, when m is too small, the gradient signal may be weak at the beginning
[Figure] [Figure]
[Figure] [Figure]
[Figure]
of the optimization
, leading to no improvement in the uniformity of the representation vectors.
Then turn our attention to Figure 3e, which plots the same metrics when m is 0.80.
The curves as a whole is highly similar to Figure 1d, with the same overfitting phenomenon.
It is also intuitive, since an excessively large margin leads the optimization process exceptionally difficult, thus preventing the “gradient dissipation” phenomenon from occurring during the optimization process.
To summarize, the above phenomenon further illustrates that the timing of gradient dissipation is very important; too early “gradient dissipation” will result in under-tuned models, while too late “gradient dissipation” will result in degraded model performance and overfitting of training.
Further, in Appendix E, we study the relation between L mpt and the Mixup-based approaches Kalantidis et al.
Zhang et al.
(2022b).
D ISCUSSION
In this section, we share some thoughts and observations to explain why we believe that the “gradient dissipation” plays a key role in SRL.
We start by considering it in the relation to the STS tasks.
For consistency with the retrieval scenario in practice, the STS tasks care about the relative order among sentence distances rather than the absolute value Reimers & Gurevych (2019).
Therefore, in the ideal case, we should let the samples closer to the anchor if they are more semantically similar to the anchor, and does not need to care about the absolute distance to the anchor.
Intuitively, the property of “gradient dissipation” only works to puts negative samples further away from the anchor compared to positive samples, which is consistent with the need for STS tasks.
On the other hand, alignment and uniformity only describe the relation between the anchor and its samples, while ignoring the relations between the positive samples and the negative samples.
In other words, this optimization form overemphasizes the characteristics of the individual sentence itself and ignores the actual connections between the semantics.
In early exploratory trials, we find that the distance gap between negative sample-anchor and positive sample points is continuously enlarged when the decouple forms are adopted for optimization (Figure 3b), which is far beyond the level when optimized by contrastive loss.
Considering the limitations of data augmentation to generate positive samples and the noise in negative samples, we suspect that it compromises the original semantic information of the pretrained models to some extent.
Combining on the above thoughts and observations, we hypothesize that the “gradient dissipation” property is responsible for the better performance in the STS evaluation.
R ELATED W ORK
Sentence representation learning (SRL) Kiros et al.
; Conneau et al.
Reimers &
Gurevych (2019); Li et al.
(2020) is one of the fundamental tasks in NLP, aiming at learning semantically rich high-dimensional representation at the sentence level.
The good sentence representations need to satisfy both (1) high correlation with human-rated similarities (intrinsic evaluation) and (2) good transferability (extrinsic evaluation)
Chiu et al.
; Faruqui et al.
The pretrained language models Peters et al.
; Devlin et al.
Liu et al.
(2019) were once regraded as the source for obtaining universal sentence representations, and the sentence representations obtained by pretrained models have shown extraordinary performance on tranfer tasks.
However, these sentence representations obtained from the pretrained models even get lower performance than the average Glove embeddings Pennington et al.
(2014) on semantic textual similarity (STS) tasks Reimers & Gurevych (2019).
Then more studies have found the word representation space of pretrained models, such as BERT Devlin et al.
and ELMo Peters et al.
(2018), are anisotropic Ethayarajh (2019), where word embeddings are concentrated on a high-dimensional conical space Gao et al.
Early works Li et al.
Su et al.
(2021) try to diminish the anisotropy of the pretrained representation space using the whitening transformation Su et al.
(2021) or the flow function Li et al.
These approaches are easy to implement, but limited improvement for STS tasks.
Recently, contrastive learning methods based on instance discrimination Zhang et al.
(2022b;c;a) are introduced to SRL.
However, a great deal of works Yan et al.
; Gao et al.
; Zhou et al.
Zhang et al.
(2022b) focus on how to obtain or generate better positive and negative samples.
At the same time, we find that even though there are significant differences in current contrastive learning methods in VRL and SRL, such as differences in pretraining and fine tuning, and differences in negative sample sizes (see Section 1 for details), few works have dabbled in the optimization mechanisms of contrastive learning in SRL.
Instead, a large number of findings from VRL, such as alignment and uniformity Wang & Isola (2020); Gao et al.
(2021), momentum updates He et al.
; Wu et al.
and bootstrap Grill et al.
(2020); Cao et al.
(2022), have been directly applied.
Although there are a large number of works Gao et al.
; Zhang et al.
(2022b;c;a)
; Klein &
Nabi (2022) in SRL that use alignment and uniformity as evaluation metrics, to our knowledge, this work first study their inadequacy as the optimization objectives in SRL.
C ONCLUSION
In this paper, we focus on the performance gap between contrastive loss and its decoupled forms, i.e., alignment and uniformity in SRL.
Our series of new findings contribute to a deeper understanding of how contrastive loss can improve the quality of sentence representation on STS tasks: (1) Alignment and Uniformity Loss or their equivalent are not suitable as alternative loss functions for contrastive loss in SRL due to their lower performance and overfitting problem; (2) The “gradient dissipation” property of contrastive loss under a small number of negative samples plays a key role in the performance on STS tasks and preventing overfitting; (3) The “gradient dissipation” property works to control a suitable distance gap of anchor-positive and anchor-negative samples, while alignment and uniformity are the properties that control the absolute distance of them;
(4) Other two loss functions with “gradient dissipation” property also can solve the overfitting problem, and obtain the same or even better performance than contrastive loss, even if their “gradient dissipation” properties are not sensitive to the number of negative samples.
We hope these findings to build a better understanding to the key properties of contrastive loss in
SRL.
Further, some new loss functions that are not bound to the form of contrastive loss can be designed via these properties, improving the quality of the sentence representation synthetically.
A E XPERIMENT D ETAILS A. E XPERIMENT S ETUP
Following recent works in SRL Gao et al.
(2021); Yan et al.
; Zhang et al.
(2022b)
, we use the SentEval toolkit Conneau & Kiela (2018) to evaluate the quality of sentence representations
For intrinsic evaluation i e., correlation to human judgments we use seven Semantic Textual Similarity (STS) datasets (STS12-16 Agirre et al.
(2012; 2013; 2014; 2015; 2016), STS-B Cer et al.
(2017) and SICK-R Marelli et al.
(2014) in SentEval.
For extrinsic evaluation, i e., transfer ability to downstream tasks, we also select seven downstream task datasets (MR Pang & Lee (2005), CR Hu & Liu (2004), SUBJ Pang & Lee (2004), MPQA Wiebe et al.
(2005), SST-2 Socher et al.
TREC Voorhees & Tice (2000) and MRPC Dolan & Brockett (2005)) to verify the migration ability of the sentence representations.
We followed the default configuration recommended by SentEval for training and evaluation.
A. T RAINING D ETAILS
We validate the effectiveness of all loss functions based on the SimCSE Gao et al.
(2021) as it is effective but simple enough, so the performance is not easily influenced by other factors.
Following Gao et al.
[Table] we observe the following points during training: (1) No data in any
STS training set is used for training; (2) 1,000,000 sentences sampled from the English Wikipediaare used for training instead; (3) Spearman correlation on STS-B development set is recorded each 125 steps; (4) The checkpoints corresponding to the highest Spearman correlation will be saved for evaluation; (5) The training period is one epoch for all pretrained models.
We implement the codes using Python3.7 and Pytorch1.12.0 and experiment with the single 32G
NVIDIA V100 GPU.
[Table]
A. P ARAMETER S ETTING
For all loss functions [Table] we perform a grid search on learning rate =
{ 7e-6, 1e-5, 3e-5, 5e-5 } and batch size = { 64, 128, 256, 512 } .
For other hyperparameters in every optimization objectives [Table] we first narrow the interval via an extensive search, then the grid search is conducted in the following ranges:
• L cl and L dcl – τ = { 0 . 03 , 0 . 05 , 0 .
07 } • L a&u – λ = { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
– α = { 1 . 0 , 2 . 0 , 3 . 0 } – t ∈ [2 , 10] , step size is• L dcl + – τ ∈ [0 . 10 , 0 . 20] , step size is 0.
• L mpt – m ∈ [0 . 20 , 0 . 35] , step size is 0.
• L met – m ∈ [0 .
50] , step size is 0.
• L mat – m ∈ { 0 .
10 π, 0 .
20 π ] , step size is 0.01 π
The optimal parameters are shown in Table 2, which are adopted to report all results and plot to all the images in this paper.
A. E VALUATION P ROTOCOL
We report the performance of the sentence representation on the STS task using the Spearman’s rank correlation, which has been widely used in recent work.
Compared with the Pearson correlation,
Spearman’s rank correlation focuses on relative Ranking instead of absolute scores, which is more in line with practical retrieval applications of text similarity matching Reimers et al.
B T HE A NALYSIS ON U NIFORMITY P ART
[Figure]
[Figure] =
[Figure]
|
[Figure]
(N
[Figure] [Figure]
||
[Figure]
|| (N=
[Figure] [Figure]
||
[Figure]
|| (N=
[Figure]
[Figure]
|
[Figure]
|| (N=64)
[Figure]
|| (N=64)
[Figure] [Figure] [Figure] [Figure]
[Figure]
Similar to the main text, we study the uniformity part of the gradient norm of all the loss functions mentioned in the main text, and their formulas are shown below:
[Equation]
[Equation] [Equation] [Equation] 
[Equation]
The derivation of all the above equations can be found in Appendix C
[Equation]
and their trends with θ ii ′ - θ ij ′ are plotted in Figure 4.
The findings of the uniformity part is highly similar to those of the alignment part:
• ∥∇ neg j cl ∥ approaches ∥∇L j uniform ∥ and ∥∇ neg j dcl ∥ as the number of negative samples increases.
• When the number of negative samples is small, “gradient dissipation” exists for ∥∇ neg j cl ∥ , while ∥∇L j uniform ∥ and ∥∇ neg j dcl ∥ do not exist.
• ∥∇ neg j dcl + ∥ exists the similar “gradient dissipation” property of ∥∇ neg j cl ∥ .
• ∥∇ posmpt ∥ , ∥∇ posmet ∥ and ∥∇ posmat ∥ also exist the propety of “gradient dissipation”, but differs from ∥∇ neg j cl ∥ in the trend of the gradient norms.
C F ORMULA D ERIVATION C. T HE DERIVATION OF E QUATION
[Equation]
[Table]
[Equation]

[Equation]
+ (cid:80) Nj,j ̸
= i exp (cid:16) ˆ h
Ti ˆ h ′ j /τ (cid:17)
I −
[Equation]
i
[Equation]
i
[Equation]
= −
[Equation]
N
[Equation]
j,j ̸
= i
[Equation]
(cid:16)
[Equation]
h Ti ˆ h
′ j /τ (cid:17) (cid:16)
ˆ h ′
i − ˆ h
i − (cid:16) ˆ h ′
j − ˆ h
i (cid:17)(cid:17)
[Equation]
(cid:16) ˆ h
[Equation]
ˆ h
′ i /τ (cid:17)
+ (cid:80) Nj,j ̸
= i exp (cid:16) ˆ h
Ti ˆ h ′ j /τ (cid:17)
I
[Equation]
i ∥
[Equation]
i ∥
[Equation]
(cid:80)
[Equation]
j ̸ = i exp (cid:16) ˆ
[Equation]
ˆ h ′ j /
[Equation]
(cid:17) (cid:16)
[Equation]
i − ˆ
[Equation]
i (cid:17)
[Equation]
(cid:16)
[Equation]
′ i /
[Equation]
(cid:17)
[Equation]
(cid:80) Nj
[Equation]
j ̸ = i
[Equation]
(cid:16) ˆ h
Ti ˆ h ′ j /
[Equation]
(cid:17)
[Equation]
̸
[Equation]
i
[Equation]
h Ti ˆ h ′ j /
[Equation]
(cid:17)
[Equation]
ˆ h ′ j
[Equation]
ˆ h i (cid:17)
[Equation]
(cid:16)
[Equation]
h
[Equation]
ˆ h
′ i / [Equation] (cid:17) [Equation]
Nj
[Equation]
j ̸
[Equation]
i exp (cid:16) ˆ h
[Equation]
ˆ h ′
[Equation]
(cid:17)
[Equation]
h i ∥
h i ∥
[Equation]
(cid:80) N j
[Equation]
j ̸ = i ∇ neg j cl
[Equation]
[Table]
[Equation]
C. T HE DERIVATION OF E QUATION 8 AND
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
(cid:16)
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
(cid:13)(cid:13)(cid:16) ˆ h
[Table]
j
[Equation]
i (cid:17) (
[Table]
[Equation] [Table] ) (cid:13)(cid:13)(cid:13) [Table]
h
i
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Table] [Equation]
[Table]
ˆ h i (cid:17) (cid:80) Nj,j ̸ = i exp (cid:16) ˆ h
Ti ˆ h ′ j /τ (cid:17)
I − M h i ∥ h i ∥ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = 1 τ exp (cid:16) ˆ h
Ti ˆ h ′ j /τ (cid:17) (cid:80) N j,j ̸ = i exp (cid:16) ˆ h
Ti ˆ h ′ j /τ (cid:17) (cid:13)(cid:13)(cid:13)(cid:16) ˆ h ′ j − ˆ
h
i (cid:17) ( I − M h i ) (cid:13)(cid:13)(cid:13) ∥
h i ∥ = 1 τ exp (cos θ ij ′ /τ )
(cid:80) Nj,j ̸
= i exp (cos θ ij ′ /τ )
(cid:13)(cid:13)(cid:13) ˆ h ′ j − ˆ h
i cos θ ij ′ (cid:13)(cid:13)(cid:13)
∥ h i ∥ = 1 τ exp (cos θ ij ′ /τ ) sin θ ij ′ (cid:80) N j,j ̸
= i exp (cos θ ij ′ /τ )
∥
h i ∥ (34)
C. T HE DERIVATION OF E QUATION
[Table]
[Equation]
where the basis for the first inequality sign is
[Equation]
C.
[Equation]
met (cid:13)(cid:13)(cid:13) AND (cid:13)(cid:13)(cid:13) ∇ neg j mat (cid:13)(cid:13)(cid:13)
Consider a generic form of the optimization objective first:
[Equation]
where ρ ( ., . ) represents a function used to calculate the similarity.
Then the gradient of L mdt with respect to h i is calculated and decomposed:
[Equation]
where ∇ negmdt is only contributed by the negative sample with the closest distance to the anchor point.
By specifying ρ ( ., . ) as dot product, l 2 -norm and angle, we can get the gradient norm of L mpt , L met and L mat with respect to the part of alignment respectively:
[Equation]

[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
Similarly
[Equation]
we can derive the gradient norm of each loss function corresponding to the part of unifor- mity respectively:
[Equation]

[Equation]
ˆ h i ,
[Equation]
(cid:13)(cid:13) (cid:13)
[Equation]
h
i
[Equation]
(cid:13)(cid:13) (cid:13)and ρ (cid:16)
ˆ h
i , ˆ h ′
i (cid:17) − min j,j ̸ = i ρ (cid:16)
ˆ h
i , ˆ h ′ j (cid:17) ≥ m
[Equation]
[Equation]
h i , ˆ h
[Equation]
and ρ (cid:16) ˆ
[Equation]
i
[Equation]
ˆ h ′
[Equation]
(cid:17)
[Equation] j [Equation] ρ (cid:16)
[Equation]
(cid:17) − min
[Equation]
̸
[Equation]
i
[Equation]
(cid:16)
[Equation]
D T HE CONNECTION BETWEEN THE TWO DECOUPLE FORMS
In this paper, two important research objectives are two decoupled forms of contrastive loss.
The first one is the alignment and uniformity proposed by Wang & Isola (2020): [Equation] [Equation]
[Equation]
And the second one is the L dcl proposed by Yeh et al.
[Equation]
where the first term of equation 43 is acknowledged by DCL’s authors to be equivalent to equation 41, while the difference between the second term and L uniform is only the order of the logarithmic function and the first summation operation when the losses are calculated for all samples in the same mini-bacth.
Their lower bounds can be obtained through Jensen’s Inequality:
[Equation]
L neg dcl − log( N − 1)
= 1 M
M (cid:88)
i log
N −
N (cid:88) j,j ̸ = i
exp(ˆ h
Ti h ′ j ) /τ ≥ τM ( N − 1)
M (cid:88) i
N (cid:88) j,j ̸ = i
ˆ h
Ti h ′ j = − 2 τM ( N − 1)
M (cid:88) i
N (cid:88) j,j ̸ = i ∥
ˆ h i − ˆ h ′ j
∥ 22 + 2 τM ( N − 1) (46) where M is batch size.
Comparing equation 44 and 46, their common optimization objective is to maximize the sum of the squares of the Euclidean distance between all pairwise samples.
Further, this optimization objective corresponds to a specific form of the minimum energy problem on the hypersphere Kuijlaars & Saff (1998);
Liu et al.
(2018), which is generalized from traditional
Thomson Problem Thomson (1904) in physics.
E R ELATION WITH M IXUP - BASED APPROACHES
We focus on a class of Mixup-based methods Kalantidis et al.
; Zhang et al.
(2022b) in contrastive learning which have similar optimization objectives to L mpt .
These methods that use Mixup
Zhang et al.
(2018) to generate hard negative samples for robustness and performance improvement,
and they have been shown to be effective in VRL Kalantidis et al.
(2020) and SRL Zhang et al.
(2022b).
It should to be noted that such methods are the improvements at the sample level, while do not change the property of “gradient dissipation” in contrastive loss.
Existing works generate mixup negative samples with two methods.
The first one is generating by linearly weighting two hard negative sample representations Kalantidis et al.
If we regard ˆ h i as the anchor, the mixup negative sample can be defined as:
[Equation]
where ˜ h ′
j represents the mixup hard negative sample and λ is the weight parameter.
The second method is generating by linearly weighting the representations of a positive sample and a random negative sample Kalantidis et al.
; Zhang et al.
(2022b):
[Equation]
where λ should be less than 0.5 to avoid generating positive samples with high probability
[Equation]
Then we discuss the relation between both two generation methods with L mpt .
When M mixup negative samples are added to contrastive loss [Equation] the following inequality holds:
[Equation]
In the following we need to perform a secondary relaxation [Equation] and this process needs to be discussed on the following sub-conditions:
Condition
If ˜ h ′ k is generated using the way in equation 47, the maximum value of the N+M terms in the log function can be taken in the first N terms [Equation] and the following inequality holds:
[Equation]
Condition 2.
If ˜ h ′ k is generated using the way in equation 48 and the maximum value of the N+M terms in the log function is taken in the first N terms, the inequality 50 still holds.
Condition 2.
If ˜ h ′ k is generated using the way in equation 48 and the maximum value of the N+M terms in the log function is taken in the latter M term, the another inequality will hold:
[Equation]
This observation provides a novel perspective to explain the performance of L mpt : (1) if mixup negative samples are generated with the first method, the optimization objective of L mpt will be equated to some extent with that of L mix by appropriately adjusting m ; (2) if mixup negative samples are generated with the second method, L mix will have two different the upper bounds with different margins, τ log( N + M ) and τ 1 − λ log( N + M ) , during the training process.
However, the margin m in L mpt will no longer change during the optimization process.
Therefore, we establish the relation between the sample-level Mixup-based approaches and L mpt .
By using a appropriate m , the optimization objective of L mpt can be similar or equal to the mixupbased methods, which provides a new perspective to explain the well-performance of L mpt -like loss functions.