UNSEE : Unsupervised Non-contrastive Sentence Embeddings (2024)

Abstract: In this paper, we introduce UNSEE, which stands for Unsupervised Non-Contrastive Sentence Embeddings. UNSEE demonstrates better performance compared to SimCSE in the Massive Text Embedding (MTEB) benchmark. We begin by highlighting the issue of representation collapse that occurs with the replacement of contrastive objectives with non-contrastive objectives in SimCSE. Subsequently, we introduce a straightforward solution called the target network to mitigate this problem. This approach enables us to harness non-contrastive objectives while ensuring training stability and achieving performance improvements similar to those seen with contrastive objectives. We have reached peak performance in non-contrastive sentence embeddings through extensive fine-tuning and optimization. These efforts have resulted in superior sentence representation models, emphasizing the importance of careful tuning and optimization for non-contrastive objectives.

## introduction contrastive learning has been used quite extensively in the sentence embedding models @xcite @xcite @xcite @xcite which hace achieved remarkable results on mteb benchmark @xcite . the fundamental role of the contrastive objective is to regularize the anisotropic embedding space of language models, ultimately enabling them to function effectively as embedding models @xcite . on the contrary, non-contrastive methods have not gained widespread popularity as the primary objective for training sentence embedding models, despite demonstrating regularization efficacy in vision @xcite @xcite . this reluctance stems from the fact that non-contrastive objectives tend to perform suboptimally in comparison to contrastive objectives, particularly in the simcse @xcite setting. for example, scd @xcite showcased that barlow twins @xcite achieves only 67.57 on the stsbenchmark @xcite test set, while simcse @xcite accomplishes 76.85. additionally, we demonstrate that the observed performance drawback is not confined to barlow twins exclusively. other well-known noncontrastive methods @xcite also suffer from inferior performance. specifically, when examining the top evaluation scores in figure 2 for the stsbenchmark development set, these non-contrastive methods consistently fall short compared to simcse, which achieves an impressive score of 82.5. despite the comparatively lower performance observed when non-contrastive objectives are employed in a sentence embedding framework, their inherent characteristics, such as the lack of negative samples and the ability to prevent dimensional collapse, as demonstrated in @xcite , inspire us to delve deeper into investigating and improving the effectiveness of non-contrastive objectives. hence, we begin by presenting empirical evidence of representation collapse observed during training with non-contrastive objectives. this includes instances utilizing siamese networks, dropout as augmentation, and even those incorporating additional parametrization with mlp layers. we delve into the potential reasons behind the suboptimal performance in section 4.1. furthermore, we introduce the target network as a novel augmentation method, which empirically enhances the diversity of embeddings and effectively mitigates the collapse associated with noncontrastive objectives. subsequently, through additional finetuning and architectural refinements, detailed in section 4.2 and section 4.3, we achieve the online encoder target encoder mlp non-contrastive objective online projection model online encoder mlp non-contrastive objective projection model nlp is cool nlp is cool nlp is cool nlp is cool online encoder target encoder mlp non-contrastive objective single projection model nlp is cool nlp is cool @xcite . the online keyword is to emphasize that the model gets gradient updates. the online projection model is similar to the projection model except for the target encoder. the target encoder is an exponentially moving average of the online network. both outputs from online and target encoders pass through the same mlp layer in the online projection model. target mlp is not employed due to the nature of fine-tuning which will slightly change the newly initialized mlp layer that will potentially corrupt the embeddings. in single projection model, target embeddings do not go through the mlp layer unlike online projection model. single projection model is identical to the architecture proposed in bsl @xcite . we only use bert-base @xcite as the encoder. ## related work competitive sentence embedding models are typically built by modifying bert @xcite with diverse configurations. in the early stages of sentence embedding development, models like infersent @xcite and the universal sentence encoder @xcite predominantly relied on lstm @xcite or the transformer @xcite architecture. the conventional bert model @xcite exhibits suboptimal performance and operates at a slower pace. sentence bert, abbreviated as sbert @xcite , represents a modified version of bert that utilizes siamese or triplet networks to generate meaningful and accurate sentence embeddings. sbert improves accuracy and significantly reduces the time required to identify the most similar pair of sentences within a set of 10,000 sentences, reducing the process from 65 hours to just 5 seconds. de-spite the integration of these enhancements into bert, a fundamental question arises: why are these modifications necessary in the first place? @xcite brings attention to a concern related to bert's sentence embeddings, specifically highlighting the presence of anisotropy in the embedding space. their empirical observations reveal that the sentence embedding space lacks smoothness and is poorly defined in certain regions, posing challenges when applying cosine similarity directly. to address this issue, they propose a solution that involves transforming sentence embeddings into a gaussian distribution that is both smooth and isotropic. this transformation is achieved through the utilization of normalizing flows. the proposed flow-based generative model is trained in an unsupervised manner with the objective of maximizing the likelihood of generating bert sentence embeddings from a standard gaussian latent variable. @xcite present mirrorbert, a method that improves sentence representations through a straightforward approach of duplicating or slightly augmenting the text input, all without external supervision. these augmentations can take place either within the input space, involving actions like random span masking, or within the feature space, using techniques such as dropout. notably, dropout is not only implemented within the mlp but also leads to the deactivation of attention heads, all figure 2: the performance of various non-contrastive objectives on stsbenchmark evaluation dataset @xcite in the projection model or simcse setting. the difference between models is the number of mlp layers. mlp layer is adopted from bsl @xcite . while preserving the model's performance across various tasks. furthermore, it has been demonstrated that mirrorbert also enhances isotropy. @xcite introduce simcse, which employs conventional dropout as a means of input augmentation. by feeding a single sentence through two passes, this approach generates two distinct feature embeddings, which can be treated as similar to positive pairs, while other sentences serve as negative samples. this dropout-based approach offers a straightforward technique for creating positive-negative pairs in contrastive learning. impressively, it achieves superior performance compared to mirror-bert with only moderate modifications. the current state-of-the-art embedding models @xcite @xcite distinguish themselves by their training on exceptionally large and extensive corpora. these corpora encompass a vast amount of both unlabeled and labeled text data. the utilization of such extensive and diverse training data has played a crucial role in the impressive performance exhibited by these models in the mteb benchmark @xcite , despite their fundamental similarity to simcse. on the contrary, models such as simcse follow a significantly different paradigm, undergoing training on a relatively modest dataset consisting of just 1 million sentences. considering the substantial difference in the scale and diversity of training data, attempting direct comparisons between simcselike models and these state-of-the-art embedding models seems impractical and might not provide meaningful insights into their relative capabilities. therefore, we exclude them from our analysis. ## background in this section, we provide an extensive overview of non-contrastive representation learning and the methods that form the core of our research. ## byol, bsl and final results in our paper, we extensively examine and engage in discussions concerning non-contrastive objectives that incorporate a siamese network architecture. however, it's important to note that our most effective configuration closely resembles byol @xcite , and we have conducted training to incorporate this configuration into our results. the ultimate model we present is a variation of bsl @xcite with dropout serving as an augmentation method. throughout our experimentation, it becomes evident that non-contrastive methods consistently outperform simcse as the while mteb aims to encompass a wide range of applications for sentence embeddings, there are noticeable score discrepancies within unsee models. despite their shared objective of optimizing feature decorrelation, implicit in the case of byol, differences in their problem formulations lead to variations in scores across different subtasks. for instance, unsee-barlow excels significantly in clustering compared to other objectives. one could argue that the exclusive focus of barlow twins on minimizing feature decorrelation might make it more effective in information dissemination, resulting in superior clustering. however, vicreg's incorporation of variance and invariance aspects may pose challenges in achieving the same level of clustering performance. another question arises regarding why this performance difference doesn't extend to retrieval. one possible explanation is that retrieval requires a finer-grained spread within a subspace, a quality that other objectives (excluding barlow twins) may achieve due to their invariance objective. nonetheless, our findings collectively reinforce the notion that non-contrastive methods contribute to a notable expansion of bert's capabilities, effectively harnessing its potential to serve as a highly effective and versatile tool for generating sentence embeddings. this empirical evidence underscores the transformative role these methods play in enhancing the utility and adaptability of bert across various sentence-related tasks. ## conclusion unsee (unsupervised non-contrastive sentence embeddings) is a simple framework for noncontrastive sentence embeddings, which outperforms simcse in the massive text embedding benchmark (mteb). we address representation collapse using a simple solution called the target network, enabling stable training and achieving performance similar to contrastive objectives. our meticulous fine-tuning leads to performant sentence embedding models, showcasing the significance of thoughtful optimization in advancing noncontrastive methods for sentence representation.