Sentence Representation Learning with Generative Objective rather than Contrastive Objective (2022)

Abstract: Though offering amazing contextualized token-level representations, current pre-trained language models take less attention on accurately acquiring sentence-level representation during their self-supervised pre-training. However, contrastive objectives which dominate the current sentence representation learning bring little linguistic interpretability and no performance guarantee on downstream semantic tasks. We instead propose a novel generative self-supervised learning objective based on phrase reconstruction. To overcome the drawbacks of previous generative methods, we carefully model intra-sentence structure by breaking down one sentence into pieces of important phrases. Empirical studies show that our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods not only on the STS benchmarks, but also on downstream semantic retrieval and reranking tasks. Our code is available at https://github.com/chengzhipanpan/PaSeR.

## introduction sentence representation learning has long been a hot research topic @xcite , for its effectiveness in a variety of downstream tasks like information retrieval and question answering @xcite . although pre-trained language models (prlms) like bert @xcite have achieved overwhelming performance on various token-level tasks, they are also criticized for being unable to produce high-quality sentence-level representations. research @xcite has shown that the native sentence representation produced by the "[cls]" token of bert shows extremely poor performance on sentence evaluation benchmarks like semantic textual similarity (sts) tasks. the primary cause of these low-quality sentence representations is the lack of effective selfsupervised sentence-level training objectives. as discussed in consert @xcite , the original sentence-level pretraining objective next sentence prediction (nsp) is too weak to provide high-quality sentence representation. therefore, recent researchers are seeking other effective self-supervised sentence-level objectives @xcite @xcite . generally, self-supervised methods include both (i) generative methods, like masked language modelling (mlm), and (ii) contrastive methods, like next sentence prediction (nsp). by treating one sentence as a whole and contrasting sentence representations with each other in the same training batch, contrastive methods have been shown extremely effective in sentence representation learning in recent years. generally, contrastive methods often use various data augmentation techniques to create different views for one sentence, and align the representations of these views within the same batch. consert @xcite utilizes techniques including token shuffling, feature cutoff, etc., and provides a general contrastive learning framework for sentence representation learning. simcse @xcite suggests using different dropout masks is a simple yet more powerful augmentation technique for creating different views of one sentence. though effective, there also exist several drawbacks in contrastive methods. (i) the training procedure of contrastive methods lacks enough linguistic interpretability. what information is encoded into the sentence representation is kept unknown. (ii) as suggested in tsdae @xcite , good performance on sts tasks by contrastive methods does not ensure good performance on downstream tasks like semantic retrieval and reranking because of the obvious inductive bias. on the contrary, generative self-supervised learn-ing techniques offer researchers good interpretability and controllable training by enabling the choice of what to generate. however, although generative methods like mlm have achieved overwhelming performance in token representation learning, little effort has been put into investigating the potential of generative methods in sentence representation learning. cross-thought @xcite and cmlm @xcite are the most representative generative methods, which both leverage the contextual sentence representations to recover the masked tokens in one sentence. unfortunately, they highly depend on the contextual information, and mainly focus on the document-level corpus, thus performing unsatisfyingly in sts tasks where representations of short texts are valued. latter, tsdae @xcite proposes to use a denoising auto-encoder to recover the original sentence from the corrupted version. although ts-dae does not depend on document-level corpus anymore, it still suffers from inferior performance on the sts benchmarks. we attribute the inferior performance of existing generative methods to the wrong modeling perspective. the existing generative methods still follow the training paradigm of contrastive methods, which considers one sentence as an inseparable whole, and learn sentence representations from the inter-sentence perspective. in this paper, we novelly propose to model sentence representation learning from the intra-sentence perspective. we especially emphasize the importance of the semantic components within the sentence, i.e. phrases. we therefore present phrases-aware sentence representation (paser), which explicitly encodes the representation of the most important phrases into sentence representations. in detail, we hypothesize a good sentence representation should be able to encode and reconstruct the important phrases in the sentence when given a suitable generation signal. inspired by simcse @xcite , we provide such generation signals by our duplication and masking strategy. as shown in figure 1 , we mask out important phrases in the original sentences, and encode these masked sentences to provide signals for phrase reconstruction. experiments show that our paser achieves the sota performance on multiple single tasks in sts in both unsupervised and supervised settings, and especially better average sts performance than simcse in the supervised setting. extensive ex-a man surfing on the sea a man [mask] on the sea a man surfing @xcite [mask] surfing on the sea ## conclusion as most pre-trained language models fail to attach enough importance to sentence-level representation learning, it usually leads to unsatisfactory performance in downstream tasks when good sentence representation is right indispensable. based on investigating the intra-sentence relationship between components of sentences (important phrases) and the whole sentence representations, we propose a generative objective to align these phrases with their corresponding sentence representations. this idea leads to paser, a phrase-aware sentence representation model. as an effective alternative in sentence representation learning, our paser achieves comparable performance with strong contrastive learning baselines on sts tasks, and better performance on the downstream semantic retrieval and reranking tasks on datasets including qqp and askubuntu.