Improving Contrastive Learning of Sentence Embeddings from AI Feedback (2023)

Abstract: Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings.However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods. Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals. In this paper, we propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF).Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning. Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings.Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods.

## introduction learning sentence embeddings with rich semantics is very important for many natural language processing tasks, such as semantic matching and information retrieval. recently, pre-trained language models @xcite @xcite provide a convenient way to get sentence embeddings. however, sentence embeddings directly generated by pre-trained language models show poor performance on semantic textual similarity (sts) tasks due to the representation degeneration problem @xcite . therefore, finding ways to further improve pre-trained models to produce better sentence embeddings becomes an crucial and fundamental challenge in natural language processing. given the shortage of labeled data for sentence embedding learning, recent studies mainly focus on unsupervised methods, such as utilizing contrastive learning methods @xcite @xcite . contrastive learning can be classified into two categories @xcite : supervised contrastive learning and unsupervised contrastive learning, depending on whether additional label information is utilized to construct positive and negative sample pairs. however, the quality of positive and negative sample pairs in unsupervised contrastive learning can be difficult to ensure. recent studies also show that data augmentation strategies in unsupervised contrastive learning may introduce some bias like length information @xcite and improper negatives @xcite . while supervised contrastive learning methods can produce more accurate sample pairs by utilizing label information, such as using supervised datasets from natural language inference @xcite , it can only provide coarse-grained labels and lack fine-grained supervision signals. we aruge that these limitations of current contrastive learning methods restrict further performance enhancement of sentence embeddings. with the emergence of large pre-trained language models (llms) @xcite @xcite , researchers hope powerful llms can help human train other ai models @xcite . one way is to use llms to generate datasets using for zero-shot learning @xcite @xcite . these methods all use predefined labels and task descriptions to generate training inputs, instead of utilizing ai feedback as supervision signals. therefore, these ## understanding contrastive learning from different feedback in this section, we categorize contrastive learning methods into four categories according to their feedback sources. we summarize the details of contrastive learning from different feedback in ## methodology in this section, we first introduce our method to generate sample pairs and the training process of claif. in order to obtain high quality sentence pairs with diverse and fine-grained similarity scores, we propose a two-step sample pair generation method: sentence pair generation and semantic similarity labeling. the generation process is shown in figure 1 . we use these sample pairs to train language models like bert and roberta. then we introduce clhaif, which combines human and ai feedback in contrastive learning of sentences embeddings. ## related work recent studies about sentence embeddings mainly focuse on using additional data to further train pre-trained language models. @xcite and @xcite propose different data aug-mentation strategies for contrastive learning and achieve significant improvements using unlabeled data. chuang et al. (2022) use equivariant contrastive learning for learning better representations. zhou et al. (2022a) and wu et al. (2022) address the bias caused by construction processes of negative and positive samples. jiang et al. (2022) use different prompt templates to produce positive pairs for contrastive learning. opitz and frank (2022) use various semantic sentence features to construct fine-grained labels for sentence embedding training. impressed by the powerful capabilities of llms @xcite , researchers pay more attention to using ai feedback from llms for zero-shot and few-shot learning. @xcite ; @xcite use ai feedback from language models to enhance in-context learning and chain-of-thoughts. @xcite and @xcite generate datasets by taking labels and prompts as the input of llms and then let llms generate training samples. @xcite design a dataset generation method for sts tasks. they construct three natural language instructions based on three discrete similarity scores and then use these instructions to steer llms to construct sentence pairs. however, it is hard to use natural language to describe various similarity scores, since the similarity score is a continuous variable with values ranging from 0 to 1. ## conclusion in this paper, we first formalize four types of contrastive learning: contrastive learning from zero feedback (clzf), contrastive learning from human feedback (clhf), contrastive learning from ai feedback (claif) and contrastive learning from human and ai feedback (clhaif). then we improve contrastive learning of sentence embeddings from ai feedback and combine human feedback with ai feedback to produce better supervision signals. experimental results show that claif and clhaif can bring substantial improvements for sentence embedding learning. we hope that learning from ai feedback can shed new lights for representation learning and contrastive learning.