A Simple Angle-based Approach for Contrastive Learning of Unsupervised Sentence Representation (2024)

Abstract: Contrastive learning has been successfully adopted in VRL (visual representation learning) by constructing effective contrastive pairs. A promising baseline SimCSE has made notable breakthroughs in unsupervised SRL (sentence representation learning) following the success of contrastive learning. However, considering the difference between VRL and SRL, there is still room for designing a novel contrastive framework specially targeted for SRL. We pro- pose a novel angle-based similarity function for contrastive objective. By examining the gra- dient of our contrastive objective, we show that an angle-based similarity function incites better training dynamics on SRL than the off-the-shelf cosine similarity: (1) effectively pulling a posi- tive instance toward an anchor instance in the early stage of training and (2) not excessively repelling a false negative instance during the middle of training. Our experimental results on widely-utilized benchmarks demonstrate the ef- fectiveness and extensibility of our novel angle- based approach. Subsequent analyses establish its improved sentence representation power.

## introduction contrastive learning has achieved promising results in vrl (visual representation learning) @xcite @xcite @xcite . however, the adoption of contrastive learning in srl (sentence representation learning) has suffered from several limitations such as inherently difficult data augmentations due to a discrete nature of nlp (natural language processing) @xcite and a limited property of plms' (pre-trained language models) representation spaces @xcite @xcite . unlike earlier * co-first authors. this work was done while myeongsoo han was at hanyang university. he is now working at kt. however, indeed there are differences between srl and vrl @xcite , which suggests that consideration of the nature of srl should precede a blind adoption of vrl's success. among several points that differentiate srl, we focus on two important points: towards this end, we first investigate which component of the contrastive objective is effective for srl. by analyzing a gradient of the contrastive objective, we find that a temperature value of normalized temperature-scaled cross entropy (nt-xent) loss @xcite and a derivative of the similarity function has a correlation with a magnitude of gradient. this indicates that both of them affect training dynamics. conforming to previous works that have reported the role of temperature @xcite , and motivated by the difference between contrastive learning of srl and vrl, we focus more on exploring better similarity functions that take into account the nature of plms and srl, which have not been well-explored in previous simcse-based studies. in this regard, we design a novel angle-based similarity function for contrastive learning of unsupervised sentence representation. comparing the derivatives of the naive cosine similarity function used in simcse and the proposed angle-based function, we find an interesting property from the derivative of our angle-based functionit exponentially increases (absolute value) from 90 to 0 degrees. we expect that this property could lead to following positive impacts: (1) the angle-based approach improves the alignment during the early stages of training due to the anisotropic space of plms with smaller angles; (2) the angle-based approach mitigates the problem of inappropriate in-batch negative sampling (i.e., false negative @xcite @xcite ) during the middle of training as it does not strongly repel the negative instances with higher angle differences (see figure 1 ). under the assumption that the angle-based ap-proach can solve some issues, we propose a simple angle-based approach for contrastive sentence embedding framework (simace), which equips with the aforementioned angle-based function. we change the vanilla cosine similarity function to the angle-based function by applying an inverse function of cosine (arccosine) and adjusting its range suitable for softmax logits of contrative objective. simace outperforms the baseline sim-cse on several off-the-shelves benchmarks, with relatively small in-batch negative instances. also, simace shows more robust performance and even outperforms the baseline in a multi-task benchmark for sentence representation. in addition, we apply our novel design to recent state-of-the-art methods based on simcse and show that simply replacing the original cosine similarity function with our angle-based similarity function can improve the performance. these results demonstrate the extensibility of our work. to verify the difference between simcse and simace, and the reason for improved performance, we conduct several experimental analyses including semantic space visualization, reporting uniformity and alignment, and training dynamics in terms of angle. we found that the reason for simace's success is that the angle-based approach is appropriate especially for unsupervised srl, though it shows unprecedented results and tendencies that are not in line with prior works in vrl (wang and isola, 2020; wang and liu, 2021; zhang et al., 2021a). ## related works and preliminary unsupervised srl in srl, high-quality representation greatly correlated with human evaluations on similarities and has been proven to be effective when transferred to downstream tasks. despite the success of transformer-based plms on transfer tasks @xcite , plmsbased representations underperformed conventional static word embeddings, such as word2vec @xcite and its augmented version @xcite , particularly in sentence representation benchmark (sts tasks @xcite ). as plms turned out to have highdimensional conical space @xcite , postprocessing methods @xcite instantly tried to mitigate the problem in plms, but were limited to improving the performance. contrastive learning-based methods aim at smoothing the bottleneck of its anisotropic property, by constructing finely tailored contrastive pairs @xcite @xcite @xcite @xcite or designing an apt contrastive objective @xcite . in unsupervised contrastive learning, it mainly falls into two components in terms of achieving these goals: 1) constructing the well-crafted pairs; 2) designing an appropriate contrastive objective. most efforts have focused on constructing the former @xcite or adding auxiliary objective on contrastive loss @xcite @xcite . to the best of our knowledge, we are the first to use the angle itself as a logit in contrastive loss; traditional contrastive loss has been applied in euclidean space, but we are the first to use it in angular space, and to provide the associated mathematical reasoning and analysis. several previous works @xcite scrutinized with the angle-based estimation of similarity in the srl. while these papers seem similar in the way they try to solve the problem, there are important differences. first, the purpose of using arccosine in @xcite is to assess transfer learning tasks. secondly, the motivation behind li and li, 2024 attempts to solve for the angle itself in a supervised setting in complex space as a means of avoiding the saturation zone of the cosine function. our work aims to mitigate several problems that can arise in unsupervised contrastive srl by utilizing the derivative nature of the arccosine function. as discussed in @xcite , there is a notable phenomenon of gradient dissipation in unsupervised contrastive learning for srl at certain angles, especially at large angles around 135 degrees. while the results of our paper may be consistent with @xcite , the assumptions of li and li, 2024 are out of our intention. preliminary in unsupervised srl, simcse systematically proposed the major components for learning sentence representations, and many recent works @xcite @xcite @xcite are originated from the following framework. first, given a collection of sentences @xmath0, positive views are derived from independently passing x i to encoder twice (i.e., dropout augmentation), while negative pairs through in-batch negative sampling @xcite . secondly, they use nt-xent loss, which is based on similarity function sim(z i , z j ): ## conclusion we have proposed a novel angle-based similarity function for unsupervised contrastive learning of sentence representation, whose property delivers a more positive impact on training dynamics in srl. through extensive experiments, we have demonstrated that angle-based similarity can be a promising alternative to the traditional cosine similarity function. after finding different aspects of uniformity and alignment, we have also performed additional experiments dealing with training dynamics and visualization of semantic space to gain a deeper understanding. furthermore, we have found that our idea can be effectively plugged into the recent state-of-the-art in srl, boosting their performances. we hope that our work will be an important milestone for future research.