Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework (2022)

Abstract: Most sentence embedding techniques heavily rely on expensive human-annotated sentence pairs as the supervised signals. Despite the use of large-scale unlabeled data, the performance of unsupervised methods typically lags far behind that of the supervised counterparts in most downstream tasks. In this work, we propose a semi-supervised sentence embedding framework, GenSE, that effectively leverages large-scale unlabeled data. Our method include three parts: 1) Generate: A generator/discriminator model is jointly trained to synthesize sentence pairs from open-domain unlabeled corpus; 2) Discriminate: Noisy sentence pairs are filtered out by the discriminator to acquire high-quality positive and negative sentence pairs; 3) Contrast: A prompt-based contrastive approach is presented for sentence representation learning with both annotated and synthesized data. Comprehensive experiments show that GenSE achieves an average correlation score of 85.19 on the STS datasets and consistent performance improvement on four domain adaptation tasks, significantly surpassing the state-of-the-art methods and convincingly corroborating its effectiveness and generalization ability.

## introduction sentence representation learning has recently attracted outspread research attention. it learns vector representations for sentences, which can be subsequently utilized on a wide range of downstream tasks, including information retrieval @xcite @xcite , language understanding and evaluation @xcite @xcite . for sentence representation learning, the contrastive learning-based approaches, including supervised and unsupervised ones, have demonstrated to be the most efficient and effective @xcite @xcite . in contrastive learning, the quality of positive and negative pairs has a large impact on the overall performance @xcite . in particular, previous supervised contrastive methods usually construct sentence pairs using human-annotated natural language inference (nli) data @xcite , and outperform unsupervised approaches by a large margin @xcite . however, the state-of-the-art supervised methods usually rely on multi-source labeled data to generalize to various downstream tasks @xcite , while such large-scale annotated data across domains are not always available. recent works also attempt to leverage the resources of unlabeled sentences for better sentence representation learning. a straightforward choice is to adopt unsupervised or self-supervised paradigms, such as bert-flow @xcite , simcse @xcite . however, the performance of these methods is still far behind the supervised counterparts. another stream of work employs retrieval strategies to obtain close sentences as potential entailment pairs, and then trains a discriminator to re-label the retrieved sentence pairs @xcite . they attain satisfactory results but with two main limitations: 1) the retrieved sentences may not be the ideal entailment to the input sentences due to the limited numbers of sentences in the banking corpus, leading to lowquality synthetic sentence pairs; 2) the retrievalbased methods can only obtain entailment pairs but not contradiction ones, which can be an important source of hard negative samples. in this paper, we propose a novel semisupervised sentence representation learning framework leveraging both annotated and unlabeled corpus to address the aforementioned issues. our 1 . ## methodology the gense comprises two neural models, including a unified generator/discriminator model for sentence pair generation and quality check, and a sentence embedding model optimized with a contrastive objective. the learning schema of these two models consists of three consecutive steps, as illustrated in figure 1 . firstly, we perform joint-training of the generator/discriminator model based on nli dataset. afterwards, we take the large-scale unlabeled sentences as inputs to generate entailment and contradiction pairs with the trained generator, which are further filtered by the discriminator to keep highquality positive and negative pairs. finally, we train the contrastive learning model based on the synthesized and human-annotated sentence pairs. we formulate all tasks into a text-to-text format with a prompt as the task signal. therefore, we can build our models from the initialization of the same pre-trained text-to-text model, which leads to high modeling simplicity. ## conclusion and future work in this work, we propose a novel semi-supervised framework, gense, for sentence representation learning. we first train a unified model for generation and discrimination, which can effectively obtain high-quality synthetic positive and negative sentence pairs from open-domain unlabelled corpus. afterwards, we train a prompt-based text-totext sentence embedding model with contrastive learning on both synthesized and labeled nli data. extensive results on sts and transfer tasks validate that gense can achieve significantly better performance than current state-of-the-arts and exhibit better generalization ability. future work includes better synthesizing strategies to generate better sentence pairs and advanced designs of semisupervised sentence representation learning frameworks based on more diverse open-domain data.