Non-contrastive sentence representations via self-supervision (2024)

Abstract: Sample contrastive methods, typically referred to simply as contrastive are the foundation of most unsupervised methods to learn text and sentence embeddings. On the other hand, a different class of self-supervised non-contrastive loss functions and methods have been considered in the computer vision community and referred to as dimension contrastive. In this paper, we thoroughly compare this class of methods with the standard baseline for contrastive sentence embeddings, SimCSE. We find that self-supervised embeddings trained using dimension contrastive objectives can outperform SimCSE on downstream tasks without needing auxiliary loss functions.

## introduction text embeddings are an important tool for a variety of nlp tasks. they provide a general and compute efficient solution to problems like topic classification, document clustering, text mining and information retrieval, among others. most modern techniques to learn text embeddings rely on minimizing a contrastive loss @xcite . this requires identifying, for each example x in the training set, a positive example x + and a set of negative examples x - i associated to x. the choice of x + and x - i is one of the main factors differentiating these techniques. unsupervised methods @xcite @xcite rely on in-batch negatives for the x - i and data augmentation for x + . supervised or weakly supervised methods @xcite @xcite @xcite rely either on mining heuristics or annotated datasets to build the positive and negative pairs. for instance, a * equal contribution. alphabetical order. common choice is to use entailment and contradiction pairs respectively, as in snli @xcite and mnli @xcite . in this work, we approach the problem of learning text embedding from the point of view of which objective function to use. we consider two selfsupervised representation learning algorithms introduced in computer vision literature: barlow twins (bt) @xcite and @xcite . what sets apart these two non-contrastive methods is their nature of being dimension contrastive according to the classification of @xcite . usual contrastive methods, defined by @xcite as sample contrastive, avoid the collapse of the learned representations by penalizing similarity of the embeddings corresponding to different data points; dimension contrastive methods regularize the objective function by decorrelating the embeddings across their dimensions. both sample and dimension contrastive methods rely on data augmentation in the unsupervised setting. while good augmentation functions are known and routinely used for image data, augmentation of textual data is usually considered trickier @xcite . one of the breakthrough of simcse is the realization that using the model stochastic dropout mask to define the augmented views of the same data point is an effective choice. the main goal of this paper is to compare sentence embeddings learned through samplecontrastive and dimension-contrastive techniques and explore different augmentation strategies. we use simcse @xcite as our samplecontrastive baseline and compare it against bt and vicreg 1 . our main findings are: i) barlow twins is competitive with unsupervised simcse as a standalone objective function and outperforms it on a majority of mteb tasks with a roberta based architectures. this is partly at odds with the finding of @xcite and @xcite which include new terms in the loss with the motivation that bt alone does not get better performances than simcse. a thorough comparison of dimension and sample contrastive methods does not exist in the literature. ii) vicreg underperforms barlow twins and simcse: we find it harder to optimize it and we cannot exclude that more hyperparameter exploration and better data augmentation would lead to better results. iii) we obtain mixed results by using supervision (for instance from nli datasets) in place of data augmentation: in no case does supervision lead to better performances across all mteb downstream task categories. ## contrastive techniques all the techniques that we experiment with in the following can be described in a unified way. consider a batch of data points s n , @xmath0, . . . , n (sentences in this work). 2 the representation e n for each point is obtained through a parametrized sentence encoder (bert and roberta are what we will use in this paper): e @xmath1. in order to consider data augmentation of any type, we assume that e θ allows for a second (possibly random) parameter ϵ specifying the augmentation e ′ @xmath2, ϵ). when training e θ in the self-supervised setting we create two embeddings (views) of each point in the batch, e (a,b) n . each of them is projected to a high-dimensional space by means of a parametrized projector z n ≡ p θ (e n ). the resulting d-dimensional vectors z n are then used in the method specific loss function. simcse -our baseline for sample contrastive methods is simcse @xcite . according to the previous definitions the unsupervised version of simcse minimizes the contrastive loss barlow twins -bt @xcite ) is one of the two dimension contrastive methods we consider. each batch contributes to the loss by an amount ## methods in order to compare with @xcite , we use the same wikipedia dataset 3 they used to train the unsupervised models. for our supervised experiments we try two datasets. the first, used also by @xcite , is the set of entailment pairs from snli @xcite and mnli @xcite . only the positive pairs are used, as hard negatives cannot be incorporated in our objectives. the other is wikiauto @xcite , a set of sentences from english wikipedia aligned with their simplified english counterpart. we consider two base models for our experiments, bert-base and roberta-base. in each case the embedding e θ that we use for downstream tasks is the embedding of the [cls] token. the projector p θ for simcse is a linear layer with the same dimension as the transformer dimension, followed by tanh activation. for bt and vicreg we follow @xcite and use two linear layers with batch normalization and relu activation, followed by an additional linear layer all of dimension 8192. larger dimensions give similar results and smaller ones progressively degrade performances. the simcse models are trained with a temperature τ = 0.05, and a learning rate of 3 × 10 -5 3 the dataset can be downloaded at this link. we experiment with three basic types of augmentations for bt and vicreg. dropout: as in @xcite we apply different dropout masks to each view of the same data point; this augmentation is parametrized by the dropout probability p @xmath3.05, 0.1, 0.2}. shuffling: for both branches we select a fraction p @xmath4.05, 0.1, 0.2, 0.3, 0.5} of the input tokens and apply a random permutation. eda @xcite : we apply eda to each branch with the same parameter α = {0.1, 0.2} for synonym replacement, random insertions, random swaps, and random deletions. for each augmentation we perform a hyperparameter scan to select the best value of the remaining parameters (learning rate and the loss coefficients in eqs. 2 and 3). we measure the spearman's rank correlation on the sts-b @xcite validation set to select the best checkpoints as in @xcite . results are shown in ## results we evaluate the embedding on a variety of downstream tasks using the massive text embedding benchmark (mteb) @xcite and report both average performances on the test and our best hypertuned models from tab. 1. we display the performances of the best models for both dropout and shuffle augmentations with overall best scores in bold. we also include results from best roberta barlow twins models trained on alternative datasets underlying best scores. alignment and uniformity are also shown. evaluating on sts tasks is a common practice that we also follow to select model checkpoints. however, this has been criticized due to the lack of correlation between sts performances and downstream task performances @xcite @xcite . finally we notice that models trained on supervised datasets can outperform unsupervised methods on certain downstream tasks, but there is no clear winner. this aligns with the finding of @xcite in which single model performance on different tasks varies a lot with no single model winning across all tasks. we also report alignment and uniformity, two metrics which are commonly considered when analyzing sample contrastive embedding techniques: the standard sample contrastive objective optimizes them in the limit of infinitely many negative samples @xcite . they are shown to empirically correlate to the embedding performance on downstream tasks, but an understanding of why uniformity is needed is lacking. @xcite derives an upper bound on the error rate for classification tasks based on three metrics, alignment, divergence, and concentration. intuitively, the latter two represent how separated the centroids of the various classes are in the embedding space and how concentrated around such centroid are the representation of the augmented members of each class. @xcite show that both the infonce @xcite and bt satisfy these criteria. see appendix b for further discussions of alignment and uniformity. ## conclusion in this work, we compare sample contrastive (sim-cse) and dimension contrastive (barlow twins, vicreg) training objectives to learn sentence embeddings. our results shows how these alternative self-supervision objectives can learn good representations, performing as well as or better than those obtained from simcse. dimension contrastive techniques are largely unexplored outside computer vision literature and we hope this work could be a step towards popularizing them in the nlp community.