Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding (2022)

Abstract: Contrastive learning has become a new paradigm for unsupervised sentence embeddings.Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts.Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences.Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines.Code is available at https://github.com/lemon0830/promptCSE.

## introduction high-quality sentence embeddings can boost the performance of pre-trained language models (plms) on many downstream tasks @xcite @xcite . recent research focuses on learning sentence embeddings in an unsupervised manner due to lack of large scale labeled data @xcite @xcite . among these methods, contrastive learning has been extensively explored and achieved remarkable success @xcite @xcite . specifically, most of them construct a positive pair by operating various textual data augmentation methods, while regard two independent sentences sampled uniformly from the training data as a negative pair. in spite of effectiveness in easing the anisotropy problem, such instance-wise optimization leads to a locally smooth embedding space, and ignores semantic relevance to some extent @xcite . moreover, due to the discrete nature of language, data augmentation can change sentence semantics significantly, and thus a positive sample are possibly turned into a negative one @xcite . to alleviate the issues, we introduce the idea of prototypical contrastive learning to unsupervised sentence embeddings learning, which is proven effective to learn structural visual embedding space @xcite . the motivation lies in that when encoding sentences into the embedding space, the sentences with similar semantics cluster together around the corresponding prototype. nevertheless, the acquisition of prototypes is inefficient if we directly apply the clustering algorithms used in @xcite , due to the requirement of extra forward pass over the training set or weak correlation with semantics @xcite . this makes us wonder whether there exists a dedicated method of mining the semantic prototypes for sentence embeddings especially based on plms? to answer this question, we attempt to think from the perspective of prompt learning @xcite . intuitively, on sentence-level nlp tasks such as classification, the neural models encode and map each input to a corresponding semantic prototype in the embedding space. for example, the sentiment analysis models divide instances into semantic prototypes related to sentiment polarity. in addition, the sentence-level tasks can be solved by providing task-specific prompts to plms as a condition even without any fine-tuning @xcite @xcite . as illustrated in figure 1 , plms can directly generate reasonable label words (e.g., "positive") for a @xmath0, when fed a prompt-wrapped sequence (e.g., "<@xmath1. thus, we argue that the representations of the "[mask]" token derived by task-specific templates can be viewed as virtual semantic prototypes, which can be obtained without using label information @xcite and clustering algorithms @xcite . besides the commonlyused templates, we manually convert each template to its negation, and use them to induce negative prototypes. back to figure 1 , with the template "<@xmath2, the word "bad" can be derived from plms. in this paper, we propose conpvp (contrastive learning with prompt-derived virtual semantic prototypes) (conpvp) for unsupervised sentence representation learning. specifically, given an input sentence, we generate the positive and negative prototypical embeddings by using a task-specific template and its negative counterpart, respectively. we use the contrastive loss to enforce the sentence embedding to be close to its positive prototype, and far apart from the negative prototype as well as the prototypes of other sentences. as illustrated in figure 2 , the issue of local smoothness can be alleviated by exploiting the semantic regularization induced by task-specific prompts, and the sentences with similar semantics are closer. we empirically evaluate our proposed conpvp on a range of semantic textual similarity tasks, and the experimental results show the substantial improvements compared with strong baselines. further,the extensive analysis and applications to transfer and clustering tasks confirm the effectiveness and robustness of our conpvp. ## method in this section, we elaborate the proposed conpvp, a novel contrastive learning approach implicitly encoding semantic structure into the embedding space. as illustrated in figure 3 , conpvp is based on the popular simcse framework @xcite and further leverages the concept of semantic prototypes. ## experiments to verify the effectiveness of our proposed method, we conduct experiments and empirical analysis on semantic textual similarity (sts) tasks under the unsupervised setting. ## application to transfer learning tasks we evaluate the quality of the sentence embeddings learned by conpvp on transfer learning tasks, in- ## application to clustering tasks we follow @xcite to consider 8 benchmark datasets for short text clustering, including searchsnippets (ss) @xcite , stackoverflow (so) @xcite , biomedical (bio) @xcite , agnews (ag) @xcite , tweet @xcite and google-news (g-t, g-s, g-ts) @xcite . we follow default settings of @xcite and use bert-base and bert-large as the backbones. we run k-means @xcite on the sentence embeddings and report the clustering accuracy averaged over 10 independent runs. as illustrated in ## conclusion in this work, we take the first step to explore the prototypical contrastive learning on unsupervised sentence embedding learning, and consider more semantic views for each instance than the recent instance-wise contrastive methods. in particular, we make use of the prompting in plms to generate the positive and negative prototypical embeddings with task-specific templates. the experiments and extensive analysis validate the effectiveness and robustness of our conpvp.