Alleviating Over-smoothing for Unsupervised Sentence Representation (2023)

Abstract: Currently, learning better unsupervised sentence representations is the pursuit of many natural language processing communities. Lots of approaches based on pre-trained language models (PLMs) and contrastive learning have achieved promising results on this task. Experimentally, we observe that the over-smoothing problem reduces the capacity of these powerful PLMs, leading to sub-optimal sentence representations. In this paper, we present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples negatives from PLMs intermediate layers, improving the quality of the sentence representation. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting, which can be seen as a plug-and-play contrastive framework for learning unsupervised sentence representation. Extensive results prove that SSCL brings the superior performance improvements of different strong baselines (e.g., BERT and SimCSE) on Semantic Textual Similarity and Transfer datasets

## introduction learning effective sentence representations is a long-standing and fundamental goal of natural language processing (nlp) communities @xcite @xcite @xcite , which can be applied to various downstream nlp tasks such as semantic textual similarity @xcite @xcite @xcite @xcite and information retrieval @xcite . compared with supervised sentence representations, unsupervised sentence representation learning is more challenging because of lacking enough supervised signals. ## background in this section, we first review the formulation of the over-smoothing issue in plms from the perspective of the intra-layer and inter-layer. then we discuss the difference of over-smoothing and annisotropy problems. ## methodology in this section, we first introduce the traditional contrastive methods for learning unsupervised sentence representation. then, we describe the proposed method sscl for building negatives and briefly illustrate how to extend sscl of other contrastive frameworks. ## analysis in this section, we first conduct qualitative experiments via probing tasks to analyse the structural of the resulting representations ( ## conclusion in this paper, we explore the over-smoothing problem in unsupervised sentence representation. then, we propose a simple yet effective method named sscl, which constructs negatives from plms intermediate layers to alleviate this problem, leading better sentence representations. the proposed sscl can easily be extended to other state-of-theart methods, which can be seen as a plug-and-play contrastive framework. experiments on seven sts datasets and seven transfer datasets prove the effec-tiveness of our proposed method. and qualitative analysis indicates our method improves the resulting model's ability of capturing the semantic and surface. also quantitative analysis shows the proposed sscl not only reduces redundant semantics but also fasts the convergence speed. as an extension of our future work, we will explore other methods to improve the unsupervised sentence representation quality.