PCL : Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings (2022)

Abstract: Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multi-augmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings.

## introduction sentence embedding learning, which aims at deriving semantically meaningful fixed-sized vectors for sentences, is a natural language processing (nlp) technique of great significance, especially for time-sensitive downstream tasks @xcite . recently, contrastive learning (cl) is proven effective to learn representation @xcite @xcite and the key to unsupervised contrastive learning for sentence embedding is to augment a given anchor sentence with an effective positive instance to construct the pairs. hence, many efforts have been made to design augmentation methods by adding noises or using heuristics, which mainly fall into two categories in terms of augmentation formatdiscrete and continuous. the former operates directly on words or n-grams in the sentence, e.g., synonym substitution @xcite , shuffling and word deletion @xcite . the latter operates on latent embeddings derived by neural encoder(s), e.g., simcse with twice dropouts @xcite . however, these existing approaches usually depend on a mono-augmenting format (i.e., either discrete or continuous) with a limited number of augmenting strategies, which suffer from learning shortcuts @xcite figure 1 : the main idea of pcl. circles and triangles denote contrastive instances encoded by f θ and f θ ′ , respectively. the red ones denote ineffective positives, blue ones denote effective positives, and gray ones denote negatives. (a) conventional cl on ineffective positive instances causes shortcut learning towards mono-augmenting biases. (b) pcl improves the probability of 'at-least-one' effective positives and performs contrasts among peer positives and cooperation between peer networks. pcl maintains two networks, and each network learns from its peer network to achieve a common agreement. (c) cooperative learning objective considers peer-positive contrasts to achieve pcl with diverse augmentations. the top panel illustrates the consistency of embedding space, while the bottom panel illustrates the agreement between the peer networks. to prevent the learning shortcuts caused by the potential biases from the mono-augmenting strategy, one straightforward solution coming into our mind is that we can consider more diverse augmentations for a given sentence in both continuous and discrete formats. besides learning from diverse instances for inherent anti-bias ability, it can also bring a great opportunity for more effective learning. in particular, controlling the qualities of the noisy and heuristic augmentations for different sentences is almost impossible 2 . as illustrated in figure 1 (a), the resulting contrastive instances may become ineffective and even poisonous for conventional cl. nonetheless, diverse instances from various augmentation strategies can notably improve the possibility of at-least-one effective positives in the contrastive instances, so how to leverage the rich relations among the diverse augmentations for more effective cl is worth exploiting. to this end, we propose a brand-new peer-contrastive learning (pcl) with diverse augmentations, and an illustration of its overall framework is shown in figure 1 @xcite . firstly, pcl not only per-2 it may cause poisonous positive. for example, given a sentence "a dog is chasing a cat.", a possible shuffled sentence is "a cat is chasing a dog.", which is semantically different. we conduct experiments on 7 standard semantic textual similarity (sts) tasks @xcite @xcite @xcite @xcite to evaluate pcl. results demonstrate that pcl significantly outperforms state of the art on 7 sts tasks. typically, pcl achieves a 2.85% improvement over the previous best approach in the averaged spearman's correlation of 7 sts tasks in the bert base setting. pcl also outperforms previous approaches across different plms initialization and model sizes. moreover, ablation study and analysis show that the two proposed components, i.e., peer-positive contrasts and peer-network cooperation, are both capable of improving unsupervised sentence embedding learning. ## peer-contrastive learning (pcl) this section begins with a formal definition of unsupervised sentence embeddings, followed by detailed formulations of our pcl with diverse augmentations ( §2.1 and §2.2). lastly, we will elaborate on our training and inference procedure for unsupervised sentence embeddings ( §2.3). ## conclusion in this paper, we propose a brand-new contrastive learning framework, dubbed as peer-contrastive learning (pcl), to capture rich relations among diverse positive peers and highlight effective positives, which are learned by cooperative learning by peer networks. besides inherent anti-bias ability by diverse augmentations, it can learn from unsupervised corpus more effectively than vanilla contrastive in the text field. experiments show that the number and diversity of augmentations are crucial to pcl. ablation study also shows that the two components of pcl, i.e., peer-positive contrast and peer-network cooperation, are both beneficial to unsupervised cl for sentence embeddings. limitation. we also recognize that our pcl framework has its certain limitations: (i) due to peer positives and encoders, our framework needs higher (~3×, i.e., 2.5 gpu-hours for base models) computation overheads compared to vanilla cl. nonetheless, the acceptable extra overhead and same inference makes our framework still practical and scalable. (ii) as a work addressing the general shortcut learning problem in a fundamental task, the proposed pcl is only evaluated on resources in english. it can be further extend to more applications such as in low-resource or other languages. (iii) the performance of our framework relies on the choice of augmentation methods, and it is hard to strictly claim which combination of the methods is optimal except experimental verification. although we have analysed the effect of varying combinations of augmentations with extensive experiments, we can only select several widely-adopted augmentations to evaluate the general effectiveness of our framework.