SCD : Self-Contrastive Decorrelation of Sentence Embeddings (2022)

Abstract: In this paper, we propose Self-Contrastive Decorrelation (SCD), a self-supervised approach. Given an input sentence, it optimizes a joint self-contrastive and decorrelation objective. Learning a representation is facilitated by leveraging the contrast arising from the instantiation of standard dropout at different rates. The proposed method is conceptually simple yet empirically powerful. It achieves comparable results with state-of-the-art methods on multiple benchmarks without using contrastive pairs. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods.

## introduction unsupervised learning of representation (a.k.a. embedding) is a fundamental problem in nlp and has been studied extensively in the literature @xcite @xcite . sentence embeddings are essential for numerous language processing applications, such as machine translation, sentiment analysis, information retrieval, and semantic search. recently, self-supervised pre-training schemes have been successfully used in the context of transformer architectures, leading to a paradigm shift in natural language processing and understanding @xcite @xcite the idea here is to employ an auxiliary task, which enforces an additional objective during training. typically, this entails predictions based on a subset of information from the context. most objectives found effective in practice are quite simple. some successful examples of such pretext tasks are masked language model (mlm), next sentence prediction (nsp), sentence order prediction (sop), etc. (devlin et al., 2019; liu et al., 1 source code and pre-trained models are available at: https://github.com/sap-samples/ acl2022-self-contrastive-decorrelation/ 2019; @xcite . when working with unlabeled data, contrastive learning is among the most powerful approaches in self-supervised learning. the goal of contrastive representation learning is to learn an embedding space in such a manner that similar sample pairs (i.e., positive pairs) stay close to each other. simultaneously, dissimilar sample pairs (i.e., negative pairs) are far pushed apart. to this end, different augmented views of the same sample and the augmented views from different samples are used as positive and negative pairs. these methods have shown impressive results over a wide variety of tasks from visual to textual representation learning @xcite @xcite . different techniques have been proposed for the augmentation and selection of positive and negative pairs. for example, declutr @xcite proposes to take different spans from the same document as positive pairs, while ct @xcite aligns embeddings of the same sentence from two different encoders. cert @xcite applies the back-translation to create augmentations of original sentences, and is-bert @xcite maximizes the agreement between global and local features. finally, clear @xcite employs multiple sentence-level augmentation strategies to learn a sentence representation. despite the simplicity of these methods, they require careful treatment of negative pairs, relying on large batch sizes @xcite or sophisticated memory strategies. these include memory banks @xcite or customized mining strategies @xcite to retrieve negative pairs efficiently. in nlp specifically, the endeavor of "hard negative mining" becomes particularly challenging in the unsupervised scenario. increasing training batch size or the memory bank size implicitly introduces more hard negative samples, coming along with the heavy burden of large memory requirements. in this paper, we introduce scd, a novel algorithm for self-supervised learning of sentence embedding. scd achieves comparable performance in terms of sentence similarity-based tasks compared with state-of-the-art contrastive methods without, e.g., employing explicit contrastive pairs. rather, in order to learn sentence representations, the proposed approach leverages the self -contrast imposed on the augmentations of a single sample. in this regard, the approach builds upon the idea that sufficiently strong perturbation of the sentence embedding reflects the semantic variations of the sentence. however, it is unclear which perturbation is simply a slight variation of the sentence without changing the semantic (positive pair) and which perturbation sufficiently modifies the semantic to create a negative sample. such ambiguity manifests itself in the augmented sample sharing the characteristics of both negative and positive samples. to accommodate this, we propose an objective function consisting of two opposing terms, which acts on augmentations pairs of a sample: i) self-contrastive divergence (repulsion), and ii) feature decorrelation (attraction). the first term treats the two augmentations as a negative pair pushing apart the different views. in contrast to that, the second term attends to the augmentations as a positive pair. thus, it maximizes the correlation of the same feature across the views, learning invariance w.r.t. the augmentation. given the opposing nature of the objectives, integrating them in a joint loss yields a min-max optimization scheme. the proposed approach avoids degenerated embeddings by framing the representation learning objective as an attraction-repulsion trade-off. simultaneously, it learns to improve the semantic expressiveness of the representation. due to the difficulty of augmentation in nlp, the proposed approach generates augmentation "on-the-fly" for each sample in the batch. to this end, multiple augmentations are produced by varying dropout rates for each sample. we empirically observed that scd is more robust to the choice of augmentations than pairwise contrastive methods; we believe that not relying on contrastive pairs is one of the main reasons for this, an observation also made in self-supervised learning literature such as byol @xcite . while other methods take different augmentation or different copies of models, we utilized the different outputs of the same sentence from standard dropout. most related to our paper is @xcite , which considers using dropout as data augmentation in the context of contrastive learning. a key novelty of our approach is that we use the dropout for creating the self-contrastive pairs, which can be utilized as both positive and negative. at last, we note that our model is different from the pairwise feature decorrelation or whitening in @xcite @xcite , which encourage similar representations between augmented views of a sample while minimizing the redundancy within the representation vector. a key difference compared to these methods is that they ignore the contrastive objective completely. in contrast, our method takes it into account and provides the means to treat self-contrastive views as positive and negative pairs simultaneously. our contribution: i) generation of sentence embeddings by leverage multi-dropout ii) elimination of reliance on negative pairs using self-contrast, iii) proposing feature decorrelation objective for non-contrastive self-supervised learning in nlp. ## method our approach relies on the generation of two views a and b of samples. to this end, augmentations are generated in embedding space for each sample x i in batch x. batches are created from samples of set @xmath0, where n denotes the number of sample (sentences). augmentations are produced by an encoder f θ , parametrized by θ. the output of the encoder is the embeddings of samples in x denoted as h @xmath1. here t denotes the embedding space. next, we let, h @xmath2. the augmentation embeddings produced per sample are then denoted h a i and h b i . to obtain the different embedding, we leverage a transformer language model as an encoder in combination with varying dropout rates. specifically, one augmentation is generated with high dropout and one with low dropout. this entails employing different random masks during the encoding phase. the random masks are associated with different ratios, r a and r b , with r @xmath3. integrating the distinct dropout rates into the encoder, we yield given the embeddings, we leverage a joint loss, consisting of two objectives: schematic illustration of the proposed approach (best shown in color). starting from an input sentence (left), two embeddings are produced by varying the dropout-rate in the encoder. patches within the encoder indicate masking due to dropout. different dropout rates and resulting embeddings color-coded: low dropout, high dropout. ## conclusion & future work we proposed a self-supervised representation learning approach, which leverages the self-contrast of augmented samples obtained by dropout. despite its simplicity, it achieves comparable results with state-of-the-arts on multiple benchmarks. future work will deal with sample-specific augmentation to improve the embeddings and, particularly, the representation alignment.