Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning (2024)

Abstract: In this paper, we proposes a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; i.e. , a higher temperature causes overfitting of the uniformity while improving the alignment in earlier phase of training. Then, we design a temperature cool-down technique based on this observation, which helps PLMs to be more suitable for contrastive learning via preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensiblity of our method.

## introduction one of the most important breakthroughs in unsupervised representation learning is the introduction of contrastive learning into the field of deep learning @xcite . in the past few years, a number of studies have sought to analyze the success of contrastive learning. for example, optimizing contrastive learning can satisfy two different properties of representations on the hypersphere, which are asymptotically quantified by the uniformity and alignment loss (the former leads to a uniformly distributed representation space and the latter makes a positive instance closer to an anchor @xcite ). these approaches have also been widely adopted in the srl (sentence representation learning) literature, where simcse @xcite successfully implemented the framework for unsupervised con-trastive learning by constructing a straightforward dropout-based positive pair. there has been a steady increase of interest in the role of a temperature (τ ) used in nt-xent loss (normalized temperature cross-entropy loss) @xcite . for example, a temperature is inversely proportional to uniformity by controlling the strength of the penalty on negative samples @xcite . also, a higher temperature can lead to a collapse @xcite , i.e., degeneration solution of representation learning @xcite . however, most studies have focused only on vrl (visual representation learning), and little information is known about the role of temperature especially for srl. in addition, there are several differences between the two fields; i.e., the number of batch size (smaller in srl), the usage of plms (pretrained language models)), and a temperature value (relatively lower in srl). in our study, we first investigate the role of temperature in simcse. interestingly, we find that the higher temperature in the earlier phase of training shows lower alignment and higher uniformity loss, indicating that higher temperature alleviates the excessive repelling of negative instances that are too close to the anchor due to the anisotropic space of plms; i.e., feature vectors form a narrow cone-like representation space @xcite @xcite . theoretically, nt-xent loss with higher temperature will degenerate to the vanilla contrastive loss, which repels every negative sample with equal strength @xcite . we assume that this can be effective for srl different from typical vrl works whose models' parameters are initialized by normal distribution foot_0 and trained from scratch. based on the above motivation, we propose temperature cool-down, a simple technique specially designed for unsupervised srl. we set a higher temperature in the first few steps of earlier training, and then cool down the temperature to the original value. the higher temperature can mitigate the phenomenon where, due to the anisotropic nature of plms' representation spaces, a smaller temperature in the early phase of training leads to unintended pulling and pushing of instances because of their excessive proximity to the anchor. in this way, temperature cool-down makes the plms' representation spaces better suited for dropout-noise based contrastive learning. empirically, our temperature cool-down improves simcse's performance on the unsupervised sentence representation benchmarks. it also has the extensibility to be used in different srl methods based on simcse. ## conclusion we explore a simple, yet tricky, technique to control the temperature value of vanilla contrastive loss, which is widely used in the srl literature. motivated by previous studies in vrl and our empirical