Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning (2022)

Abstract: Contrastive learning has been demonstrated to be effective in enhancing pre-trained language models (PLMs) to derive superior universal sentence embeddings. However, existing contrastive methods still have two limitations. Firstly, previous works may acquire poor performance under domain shift settings, thus hindering the application of sentence representations in practice. We attribute this low performance to the over-parameterization of PLMs with millions of parameters. To alleviate it, we propose PromCSE (Prompt-based Contrastive Learning for Sentence Embeddings), which only trains small-scale Soft Prompt (i.e., a set of trainable vectors) while keeping PLMs fixed. Secondly, the commonly used NT-Xent loss function of contrastive learning does not fully exploit hard negatives in supervised learning settings. To this end, we propose to integrate an Energy-based Hinge loss to enhance the pairwise discriminative power, inspired by the connection between the NT-Xent loss and the Energy-based Learning paradigm. Empirical results on seven standard semantic textual similarity (STS) tasks and a domain-shifted STS task both show the effectiveness of our method compared with the current state-of-the-art sentence embedding models.

## introduction learning universal sentence embeddings @xcite @xcite @xcite which convey high-level semantic information without task-specific fine-tuning is a vital research problem in natural language processing (nlp) communities. it could benefit a wide range of applications such as information retrieval, question answering, etc @xcite . recently, fine-tuning pre-trained language models (plms) @xcite with contrastive learning, which aims to pull semantically close samples together and push apart dissimilar samples, has achieved extraordinary success in learning universal sentence representations @xcite @xcite @xcite . in these works, positive pairs are formed via data augmentation or supervised datasets, whereas negative pairs are derived from different sentences within the same minibatch. then contrastive learning objective like normalized temperature-scaled cross-entropy loss (nt-xent) @xcite is used for optimizing the model parameters. as a typical example, simcse @xcite uses the standard dropout as augmentation for constructing positive pairs and achieves extraordinarily strong performance on seven standard semantic textual similarity (sts) tasks. though effective, existing contrastive methods for learning sentence representations still have two limitations. firstly, since universal sentence embeddings are often trained on a large corpus and used off-the-shelf on a diverse range of tasks, such domain shifts are commonplace and may pose challenges to the performance. as figure 1 shows, simcse's performance drops significantly when applied on a domain-shifted sts task @xcite , where the texts are image captions. such non-robustness of large plms towards domain shifts has also been observed in other studies. @xcite ; @xcite found that tuning plms with millions of parameters may result in overfitting to the training data distribution and hence vulnerability to domain shifts. secondly, the commonly used nt-xent loss function in supervised sentence embedding models does not fully exploit the hard negatives. moreover, recent studies @xcite have shown that the softmax-based loss is insufficient to acquire the discriminating power. thus, nt-xent loss in supervised models may not separate positives and hard negatives sufficiently. in this paper, we propose two techniques to address the above-mentioned limitations. firstly, we propose the prompt-based contrastive learning for sentence embeddings (promcse) to alleviate the domain shift problem, inspired by prompt tuning @xcite . specifically, we modify simcse by freezing the entire pre-trained model and add multi-layer learnable soft prompt, which is simple yet achieves a good balance between the expressiveness and the robustness to distributional changes. secondly, we show that the contrastive learning framework under nt-xent loss @xcite could be seen as an instance of energy-based learning @xcite @xcite . inspired by this connection, we propose an energybased hinge (eh) loss to supplement nt-xent loss under supervised settings, which enhances the pairwise discriminative power by explicitly creating an energy gap between positive pairs and the hardest negative pairs. we performed extensive experiments using the seven commonly used sts tasks and another out-of-domain sts task. for the same-domain setting, the unsupervised promcse can outperform simcse by around 2.2 points and is on par with the current state-of-the-art (sota) sentence embedding method on the seven standard sts tasks. for the out-of-domain setting, the proposed unsupervised promcse can achieve 3.7 absolute points improvements over simcse and even 1.2 absolute points improvements over the current sota method, which demonstrates its robustness to domain shifts. moreover, we also demonstrate that the eh loss can improve super-vised simcse and promcse consistently over multiple pre-trained backbone models, achieving stateof-the-art results among supervised sentence representation learning methods. our contributions are summarized as follows: ## methodology in this section, we first present promcse, a promptbased contrastive learning framework for both un-supervised and supervised sentence representation learning in section 3.1. then we demonstrate that the contrastive learning framework under nt-xent loss is an instance of energy-based learning in section 3.2. eventually, inspired by energy-based learning, we design an energy-based hinge loss to supplement nt-xent loss when hard negatives are available in supervised datasets in section 3.3. ## experiments our experiments are composed of two parts. we first verify the effectiveness of our proposed approach on seven standard sts tasks in section 4.1. then we evaluate the domain shift robustness of our approach by testing on a domain shift sts task in section 4.2. ## ablation studies we investigate how different ways of choosing prompt type, prompt length and margin m affect our models. we use bert base model to evaluate on seven standard sts tasks and the cxc-sts task. type of soft prompt in promcse, we prepend multi-layer soft prompts to plms instead of only the input (embedding) layer as @xcite . ## alignment and uniformity analysis alignment and uniformity are two properties proposed by @xcite to measure the quality of representations. specifically, given the distribution of positive pairs p pos and the distribution of the whole dataset p data , alignment computes the expected distance between normalized embeddings of the paired sentences: model align uniform bert base (first-last avg.) 0.195 -1.304 unsup-simcse-bert base 0.238 -2.337 unsup-promcse-bert base 0.117 -1.354 sup-simcse-bert base 0.241 -3.246 sup-simcse-bert base + eh 0.260 -3.349 sup-promcse-bert base 0.325 -3.268 sup-promcse-bert base + eh 0.366 -3.397 while uniformity measures how well the embeddings are uniformly distributed in the representation space: ## conclusion this paper presents promcse, a prompt-based contrastive learning framework that improves universal sentence embeddings for resilience to domain shifts. additionally, we theoretically show that the contrastive learning framework under nt-xent loss is an instance of energy-based learning. to further boost the performance of supervised sentence embeddings, we propose an energy-based hinge loss to supplement nt-xent loss. extensive experiments on seven sts tasks and one domain shift sts task both verify the effectiveness of our method compared to current state-of-the-art supervised and unsupervised sentence embedding models.