PAUSE : Positive and Annealed Unlabeled Sentence Embedding (2021)

Abstract: Sentence embedding refers to a set of effective and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural language processing (NLP) applications. The majority of these techniques are either supervised or unsupervised. Compared to the unsupervised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sentence pairs, which is not available in many industrial scenarios. To that end, we propose a generic and end-to-end approach – PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of-the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the burden of extensive manual annotation work.

## introduction a sentence embedding is a numerical representation used to describe the meaning of an entire sentence. embeddings of this type are becoming increasingly important for many downstream tasks in the language understanding domain, such as similarity or sentiment analysis. some earlier methods, like glove @xcite , bert @xcite and roberta @xcite pool directly from underlying token-level embeddings to create a sentence representation. recently, these pooling strategies have been challenged by various parameterized policies that can be optimized on domain specific tasks. the majority of these are either unsupervised or supervised. while unsupervised methods only utilize unlabeled sentences, supervised methods can quickly customize the embeddings by using domain specific labels. as a consequence, supervised methods make less assumptions about optimization objectives and usually achieve better results. however, supervised training requires a large amount of labeled sentence pairs, which is usually unavailable. in many real scenarios, the dataset turns out to be positiveunlabeled (i.e. pu dataset), where the majority is unlabeled and the rest of the samples are labeled as positive. the methods that enable learning binary classifiers on pu datasets are called pu learning. to bridge the gap between supervised and unsupervised approaches, we incorporate state-of-the-art pu learning with the general supervised sentence embedding approaches, proposing a novel method -pause (positive and annealed unlabeled sentence embedding) foot_0 . the main highlights of pause include: ## related work among unsupervised sentence embedding methods, some are capable of exploring the relations among sub-sentences, such as skip-thoughts @xcite , fastsent @xcite , quick-thoughts @xcite and discsent @xcite . these methods assume that adjacent sentences always have similar semantics. however, not every corpus is long enough, perfectly ordered or coherent enough to fulfill that assumption, which limits their applicable domains. other unsupervised methods merely focus on the internal structures within each sentence, such as paragraphvectors @xcite , doc2vecc @xcite , sent2vec @xcite , wmd @xcite , gem @xcite and is-bert @xcite . in general, those unsupervised approaches optimize objectives based on assumptions, which limits their embeddings from being adapted towards different applications. recently, several concurrently proposed methods, such as @xcite @xcite , adopt contrastive objectives by constructing different views from the same sentence. @xcite achieved superior results by simply using dropout to create different views. the supervised approaches, on the other hand, are usually (1) trained in an end-to-end manner, (2) following a dual encoder architecture and (3) finetuned from a model pretrained on snli (stanford natural language inference) @xcite and multi-genre nli @xcite datasets. nli is the task of determining whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given a premise 2 . the recent representative methods include infersent @xcite , use (universal sentence encoder) variants @xcite , sbert (sentence-bert) @xcite and labse (language-agnostic bert sentence embedding) @xcite . built upon pretrained models, they can effectively learn good embeddings from the labeled sentence pairs. however, this approach is not feasible in scenarios where the quantity of annotations is limited. rather than purely labeled or unlabeled, in many real scenarios, the dataset turns out to be positiveunlabeled (pu), where a small portion of sentence pairs are labeled as positive samples and the rest are unlabeled. to address this type of problem, the gap between supervised and unsupervised methods has to be filled. levi experimented with incorporating an unsupervised regularization criteria in the supervised loss. although @xcite reported better generalization capability, all of the samples still have to be labeled. jiang et al. made an early attempt to apply pu learning -particularly for matrix factorization @xcite -to obtain word embeddings for low-resource languages. existing pu learning methods can be divided into three categories based on how unlabeled data is handled. the first category, with methods like @xcite , tries to assign labels to unlabeled data in a heuristic-driven and iterative manner which makes the training scattered in steps/phases and hard to implement in practice. the second includes methods like @xcite that treat unlabeled data as negative with lower confidence. this can be more computationally expensive to tune. the third category, with methods such as upu @xcite , nnpu @xcite , pubn @xcite and self-pu @xcite , regards each unlabeled sample as a weighted mixture of being positive and negative. this third category optimizes a so called pu loss, and has recently become dominant due to its wide applicability and end-to-end nature. however, a major limitation is that these algorithms are only applicable to binary classification problems. in this work, we show how to adapt pu learning to effectively learn sentence embeddings from multi-class pu datasets. ## the proposed method we propose a generic and end-to-end approach to obtain sentence embeddings in a setup that is a generalized natural language inference (gnli) task. this approach can have (1) any number of classes and (2) the majority of the sentence pairs unlabeled. let x be the set of sentence pairs in the entire dataset; as illustrated in figure 1 u) , where c(≥ 2) is the total number of entailment classes, x (p)(c) denotes the n (c) p sentence pairs labeled as the c-th class, and x (u) represents the n u unlabeled pairs. for the n @xmath0, the binary label towards the c-th class for the i-th pair, and the i-th unlabeled sample respectively. ## experiments inspired by previous work @xcite @xcite , we evaluate pause on sts (semantic textual similarity) and senteval foot_1 in both supervised and unsupervised settings. we also show the robustness of pause in a real industrial use case. in our experiments, we test two versions of pause: pausebase (110m parameters) and pause-small (4.4m parameters), which use uncased bert-base @xcite and bert-small @xcite foot_2 as their encoder model, respectively. pause is trained by minimizing (11) with α = 3 (searched 3, 4, and 5) using the adam optimizer with learning rate 7.5e-5 (searched 1e-3, 1e-4, 7.5e-5, 5e-5, 2.5e-5, 1e-5 and 1e-6). the experiments are carried out on a machine (managed virtual machine instance foot_3 ) with four virtual cpus (intel xeon 2.30ghz), 15gb ram, and four gpus (nvidia tesla p100). since pause requires a large batch size to ensure enough labeled samples from each class in every mini-batch, we use a batch size of 128 and 1,024 for pause-base and pause-small respectively, fully utilizing the gpu capacity. ## conclusions and future work in this work, we attempt to bridge the gap between supervised and unsupervised sentence embedding techniques, proposing pause -a generic and endto-end sentence embedding approach that exploits the labels and explores the unlabeled sentence pairs simultaneously. pause trained on nli datasets achieves state-of-the-art results on unsupervised sts tasks, and also performs well on many downstream domain-specific tasks. in all of our experiments, we observe that pause keeps performing well with a reduced number of labeled samples, as long as more than 5-10% of the dataset is labeled. this indicates that pause is a label-efficient sentence embedding approach that can be effectively applied to datasets where only a small part is labeled while the rest remains unlabeled. we also demonstrate that pause helps lower the labeling requirement for an industrial use case aimed at encoding company descriptions. in that sense, pause pushes the application boundary of sentence embeddings to include many more real-world scenarios where labeled samples are scarce. the possible extensions of this work include (1) augmenting the the labels with dropout, (2) experimenting with contrastive supervised loss, and (3) exploring how pause can be extended with con-textual sentence embeddings.