D e CLUTR : Deep Contrastive Learning for Unsupervised Textual Representations (2021)

Abstract: Sentence embeddings are an important component of many natural language processing (NLP) systems. Like word embeddings, sentence embeddings are typically learned on large text corpora and then transferred to various downstream tasks, such as clustering and retrieval. Unlike word embeddings, the highest performing solutions for learning sentence embeddings require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. In this paper, we present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Inspired by recent advances in deep metric learning (DML), we carefully design a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. When used to extend the pretraining of transformer-based language models, our approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. Importantly, our experiments suggest that the quality of the learned embeddings scale with both the number of trainable parameters and the amount of unlabelled training data. Our code and pretrained models are publicly available and can be easily adapted to new domains or used to embed unseen text.

## introduction due to the limited amount of labelled training data available for many natural language processing (nlp) tasks, transfer learning has become ubiquitous @xcite . for some time, transfer learning in nlp was limited to pretrained word embeddings @xcite 1 https://github.com/johngiorgi/declutr 2014). recent work has demonstrated strong transfer task performance using pretrained sentence embeddings. these fixed-length vectors, often referred to as "universal" sentence embeddings, are typically learned on large corpora and then transferred to various downstream tasks, such as clustering (e.g. topic modelling) and retrieval (e.g. semantic search). indeed, sentence embeddings have become an area of focus, and many supervised @xcite , semi-supervised @xcite @xcite and unsupervised @xcite @xcite @xcite approaches have been proposed. however, the highest performing solutions require labelled data, limiting their usefulness to languages and domains where labelled data is abundant. therefore, closing the performance gap between unsupervised and supervised universal sentence embedding methods is an important goal. pretraining transformer-based language models has become the primary method for learning textual representations from unlabelled corpora @xcite @xcite @xcite . this success has primarily been driven by masked language modelling (mlm). this selfsupervised token-level objective requires the model to predict the identity of some randomly masked tokens from the input sequence. in addition to mlm, some of these models have mechanisms for learning sentence-level embeddings via self-supervision. in bert @xcite , a special classification token is prepended to every input sequence, and its representation is used in a binary classifi-cation task to predict whether one textual segment follows another in the training corpus, denoted next sentence prediction (nsp). however, recent work has called into question the effectiveness of nsp @xcite @xcite . in roberta @xcite , the authors demonstrated that removing nsp during pretraining leads to unchanged or even slightly improved performance on downstream sentencelevel tasks (including semantic text similarity and natural language inference). in albert @xcite , the authors hypothesize that nsp conflates topic prediction and coherence prediction, and instead propose a sentence-order prediction objective (sop), suggesting that it better models inter-sentence coherence. in preliminary evaluations, we found that neither objective produces good universal sentence embeddings (see appendix a). thus, we propose a simple but effective self-supervised, sentence-level objective inspired by recent advances in metric learning. metric learning is a type of representation learning that aims to learn an embedding space where the vector representations of similar data are mapped close together, and vice versa @xcite @xcite . in computer vision (cv), deep metric learning (dml) has been widely used for learning visual representations @xcite @xcite @xcite @xcite @xcite @xcite . generally speaking, dml is approached as follows: a "pretext" task (often self-supervised, e.g. colourization or inpainting) is carefully designed and used to train deep neural networks to generate useful feature representations. here, "useful" means a representation that is easily adaptable to other downstream tasks, unknown at training time. downstream tasks (e.g. object recognition) are then used to evaluate the quality of the learned features (independent of the model that produced them), often by training a linear classifier on the task using these features as input. the most successful approach to date has been to design a pretext task for learning with a pair-based contrastive loss function. for a given anchor data point, contrastive losses attempt to make the distance between the anchor and some positive data points (those that are similar) smaller than the distance between the anchor and some neg-ative data points (those that are dissimilar) @xcite . the highest-performing methods generate anchor-positive pairs by randomly augmenting the same image (e.g. using crops, flips and colour distortions); anchor-negative pairs are randomly chosen, augmented views of different images @xcite @xcite . in fact, @xcite demonstrate that the mlm and nsp objectives are also instances of contrastive learning. inspired by this approach, we propose a selfsupervised, contrastive objective that can be used to pretrain a sentence encoder. our objective learns universal sentence embeddings by training an encoder to minimize the distance between the embeddings of textual segments randomly sampled from nearby in the same document. we demonstrate our objective's effectiveness by using it to extend the pretraining of a transformer-based language model and obtain state-of-the-art results on senteval @xcite ) -a benchmark of 28 tasks designed to evaluate universal sentence embeddings. our primary contributions are: ## related work previous works on universal sentence embeddings can be broadly grouped by whether or not they use labelled data in their pretraining step(s), which we refer to simply as supervised or semi-supervised and unsupervised, respectively. supervised or semi-supervised the highest performing universal sentence encoders are pretrained on the human-labelled natural language inference (nli) datasets stanford nli (snli) @xcite and multinli @xcite . nli is the task of classifying a pair of sentences (denoted the "hypothesis" and the "premise") into one of three relationships: entailment, contradiction or neutral. the effectiveness of nli for training universal sentence encoders was demonstrated by the supervised method infersent @xcite . universal sentence encoder (use) @xcite is semi-supervised, augmenting an unsupervised, skip-thoughts-like task @xcite , see section 2) with supervised training on the snli corpus. the recently published sentence transformers @xcite ) method fine-tunes pretrained, transformer-based language models like bert @xcite using labelled nli datasets. unsupervised skip-thoughts @xcite and fastsent @xcite are popular unsupervised techniques that learn sentence embeddings by using an encoding of a sentence to predict words in neighbouring sentences. however, in addition to being computationally expensive, this generative objective forces the model to reconstruct the surface form of a sentence, which may capture information irrelevant to the meaning of a sentence. quickthoughts @xcite addresses these shortcomings with a simple discriminative objective; given a sentence and its context (adjacent sentences), it learns sentence representations by training a classifier to distinguish context sentences from non-context sentences. the unifying theme of unsupervised approaches is that they exploit the "distributional hypothesis", namely that the meaning of a word (and by extension, a sentence) is characterized by the word context in which it appears. our overall approach is most similar to sentence transformers -we extend the pretraining of a transformer-based language model to produce useful sentence embeddings -but our proposed objective is self-supervised. removing the dependence on labelled data allows us to exploit the vast amount of unlabelled text on the web without being restricted to languages or domains where labelled data is plentiful (e.g. english wikipedia). our objective most closely resembles quickthoughts; some distinctions include: we relax our sampling to the encoder and pooler are trained to minimize the distance between embeddings via a contrastive prediction task, where the other embeddings in a minibatch are treated as negatives (omitted here for simplicity). (b) positive spans can overlap with, be adjacent to or be subsumed by the sampled anchor span. (c) the length of anchors and positives are randomly sampled from beta distributions, skewed toward longer and shorter spans, respectively. ## results in subsection 5.1, we compare the performance of our model against the relevant baselines. in the remaining sections, we explore which components contribute to the quality of the learned embeddings. ## discussion and conclusion in this paper, we proposed a self-supervised objective for learning universal sentence embeddings. our objective does not require labelled training data and is applicable to any text encoder. we demonstrated the effectiveness of our objective by evaluating the learned embeddings on the sente-val benchmark, which contains a total of 28 tasks designed to evaluate the transferability and linguistic properties of sentence representations. when used to extend the pretraining of a transformerbased language model, our self-supervised objective closes the performance gap with existing methods that require human-labelled training data. our experiments suggest that the learned embeddings' quality can be further improved by increasing the model and train set size. together, these results demonstrate the effectiveness and feasibility of replacing hand-labelled data with carefully designed self-supervised objectives for learning universal sentence embeddings. we release our model and code publicly in the hopes that it will be extended to new domains and non-english languages.