On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning (2023)

Abstract: Incorporating contrastive learning objectives in sentence representation learning (SRL) has yielded significant improvements on many sentence-level NLP tasks. However, it is not well understood why contrastive learning works for learning sentence-level semantics. In this paper, we aim to help guide future designs of sentence representation learning methods by taking a closer look at contrastive SRL through the lens of isotropy, contextualization and learning dynamics. We interpret its successes through the geometry of the representation shifts and show that contrastive learning brings isotropy, and drives high intra-sentence similarity: when in the same sentence, tokens converge to similar positions in the semantic space. We also find that what we formalize as “spurious contextualization” is mitigated for semantically meaningful tokens, while augmented for functional ones. We find that the embedding space is directed towards the origin during training, with more areas now better defined. We ablate these findings by observing the learning dynamics with different training temperatures, batch sizes and pooling methods.

## introduction since vanilla pre-trained language models do not perform well on sentence-level semantic tasks, sentence representation learning (srl) aims to finetune pre-trained models to capture semantic information @xcite @xcite . recently, it has gradually become de facto to incorporate contrastive learning objectives in sentence representation learning @xcite @xcite . representations of pre-trained contextualized language models @xcite @xcite have long been identified not to be isotropic, i.e., they are not uniformly distributed in all directions but instead occupying a figure 1 : expanded semantic space produced by contrastive learning (cl), visualized with umap. at the beginning of training, all embeddings occupied a narrow cone. after 200 steps of fine-tuning with a contrastive loss, they spread out to define a larger semantic space. the concept of learning dynamics focuses on what happens during the continuous progression of fine-tuning pre-trained language models. this has drawn attention in the field @xcite , with some showing that fine-tuning mitigates the anisotropy of embeddings @xcite , to different extent according to the downstream tasks. however, it is argued that the performance gained in fine-tuning is not due to its enhancement of isotropy in the em-bedding space @xcite . moreover, little research is conducted on isotropy of sentence embedding models, especially contrastive learning-based sentence representations. vanilla transformer models are known to underperform on sentence-level semantic tasks even compared to static embedding models like glove @xcite , whether using the [cls] token or averaging word embeddings in the output layer. since @xcite proposed sbert, it has become the most popular transformers-based framework in sentence representation tasks. the state-of-the-art is further improved by integrating contrastive learning objectives @xcite @xcite . the other line of works concern post-processing of embeddings in vanilla language models @xcite @xcite to attain better sentence representations. learning dynamics in fine-tuning was previously investigated, revealing isotropy shifts in the process @xcite , but few studies have systematically investigated relevant pattern shifts in sentence representation models, and none has drawn connections between these metrics and the performance gains on sentencelevel semantic tasks. while some implicitly studied this problem by experimenting on nli datasets @xcite @xcite , we argue that a more extensive study on the geometry change during fine-tuning sota sentence embedding models with contrastive objectives is neccessary. in this work, we demystify the mechanism of why contrastive fine-tuning works for sentence representation learning. 1 our main findings and contributions are as follows: our study offers fundamental insights into using contrastive objectives for sentence representation learning. with these, we aim to shed light on future designs of sentence representation learning methods. ## connecting to frequency bias the imbalance of word frequency has long been identified to be relevant to the anisotropy of trained embeddings @xcite . this has been also empirically observed in pre-trained transformers like bert @xcite . @xcite draw connection between frequency bias and the unideal performance of pre-trained language models on sts tasks, through deriving individual words as connections of contexts, concluding that rare words fail to play the role of connecting context embeddings. @xcite show that when fine-tuning pre-trained language models under the setting of siamese architecture on sts-b datasets, the frequency bias is largely removed, with less significant frequency-based distribution of embed-dings. however, it is also pointed out that these trained models are still highly anisotropic, which as we showed in section 2.5, does not hold in the context of contrastive training, which, with sufficient data, has theoretical promise toward uniformity @xcite . therefore, it is of interest to see the corresponding behaviors of frequency bias shifts in the context of contrastive learning, and more importantly, how this correlates with our surprising finding on intrasentence similarity. ## ablation analysis in this section, we provide a derivation to interpret the role of temperature in cl, inspiring the searching method of its optimal range. we also show that contrastive frameworks are less sensitive to batch size at optimal temperature for srl, unlike in visual representation learning. ## conclusion in this paper, we demystify the successes of using contrastive objectives for sentence representation learning through the lens of isotropy and learning dynamics. we showed the theoretical promise of uniformity brought by contrastive learning through measuring anisotropy, complemented by showing the flattened domination of top dimensions. we then uncovered a very interesting yet under-covered pattern: contrastive learning learns to converge tokens in a same sentence, bringing extremely high intra-sentence similarity. we then explained this pattern by connecting it to frequency bias, and showed that semantically functional tokens fall back to be the by products of semantically meaningful tokens in a sentence, following wherever they travel in the semantic space. lastly, we ablate all findings through temperature, batch size and pooling method, providing a closer look at these patterns through different angles.