H i CL : Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings (2023)

Abstract: In this paper, we propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness. Traditional methods typically encode a sequence in its entirety for contrast with others, often neglecting local representation learning, leading to challenges in generalizing to shorter texts. Conversely, HiCL improves its effectiveness by dividing the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence-level relationships. Further, considering the quadratic time complexity of transformers over input tokens, HiCL boosts training efficiency by first encoding short segments and then aggregating them to obtain the sequence representation. Extensive experiments show that HiCL enhances the prior top-performing SNCSE model across seven extensively evaluated STS tasks, with an average increase of +0.2% observed on BERT_{large} and +0.44% on RoBERTa_{large} .

## introduction current machine learning systems benefit greatly from large amounts of labeled data. however, obtaining such labeled data is expensive through annotation in supervised learning. to address this issue, self-supervised learning, where supervisory labels are defined from the data itself, has been proposed. among them, contrastive learning @xcite @xcite @xcite has become one of the most popular self-supervised learning methods due to its impressive performance across various domains. the training target of contrastive learning is to learn a representation of the data that maximizes the similarity between positive examples and minimizes the similarity between negative examples. to achieve better performance, existing methods mainly focus on designing better positive examples @xcite @xcite @xcite or investigating the role of the negative examples @xcite @xcite . despite the success, existing methods augment data at the level of the full sequence @xcite . such methods require calculating the entire sequence representation, leading to a high computational cost. additionally, it also makes the task of distinguishing positive examples from negative ones too easy, which doesn't lead to learning meaningful representations. similarly, methods like clear @xcite demonstrated that pre-training with sequence-level naïve augmentation can cause the model to converge too quickly, resulting in poor generalization. in contrast, @xcite considered modeling in-sequence (or local) relationships for language understanding. they divide the sequence into smaller segments to learn intrinsic and underlying relationships within the sequence. since this method is effective in modeling long sequences by not truncating the input and avoiding loss of information, it achieves promising results. given this success, a natural question arises: is it possible to design an effective and efficient contrastive learning framework by considering the local segmentlevel and global sequence-level relationships? to answer the question, in this paper, we propose a hierarchical contrastive learning framework, hicl, which not only considers global relationships but also values local relationships, as illustrated in figure 1 . specifically, given a sequence (i.e., sentence), hicl first divides it into smaller segments and encodes each segment to calculate local segment representation respectively. it then aggregates the local segment representations belonging to the same sequence to get the global sequence representation. having obtained local and global representations, hicl deploys a hierarchical contrastive learning strategy involving both segment-level and sequence-level contrastive learning to derive an enhanced representation. for local contrastive learning, each segment is fed into the model twice to form the positive pair, with segments from differing sequences serving as the negative examples. for global contrastive learning, hicl aligns with mainstream baselines to construct positive/negative pairs. we have carried out extensive experiments on seven sts tasks using well-representative models bert and roberta as our backbones. we assess the method's generalization capability against three baselines: simcse, esimcse, and sncse. as a result, we improve the current state-of-the-art model sncse over seven sts tasks and achieve new state-of-the-art results. multiple initializations and varied training corpora confirmed the robustness of our hicl method. our contributions are summarized below: ## preliminaries: contrastive learning in this paper, we primarily follow simclr's framework @xcite as our basic contrastive framework and describe it below. the general training objective of contrastive learning @xcite is to distinguish similar pairs from dissimilar pairs, where similar pairs are constructed using pre-defined data augmentation techniques and dissimilar pairs are other examples in the same batch. specifically, for an arbitrary example x i in a batch b, the infonce loss l g brings the representation h i closer to positive instance representation h + i and away from negative ones h benefiting from human-defined data augmentations, it can generate numerous positive and negative examples for training without the need for explicit supervision, which is arguably the key reason why self-supervised learning can be effective. positive instance designing effective data augmentations to generate positive examples is a key challenge in contrastive learning. various methods such as back-translation, span sampling, word deletion, reordering, and synonym substitution have been explored for language understanding tasks in prior works such as cert @xcite , declutr @xcite , and clear @xcite . different from previous approaches that augment data at the discrete text level, sim-cse @xcite first applied dropout @xcite twice to obtain two intermediate representations for a positive pair. specifically, given a transformers model, e θ (parameterized by θ), @xcite and a training instance x i , h @xmath0,p (x i ) and h + @xmath1,p + (x i ) are the positive pair that can be used in eq. 1, where p and p + are different dropout masks. this method has been shown to significantly improve sentence embedding performance on seven sts tasks, making it a standard comparison method in this field. negative instance negative instance selection is another important aspect of contrastive learning. simclr simply uses all other examples in the same batch as negatives. however, in dclr @xcite , around half of in-batch negatives were similar to simcse's training corpus (with a cosine similarity above 0.7). to address this issue, sncse @xcite introduced the use of negated sentences as "soft" negative samples (e.g., by adding not to the original sentence). additionally, instead of using the [cls] token's vector representation, sncse incorporates a manually designed prompt: "the sentence of x i means [mask]" and takes the [mask] token's vector to represent the full sentence x i . this approach has been shown to improve performance compared to using the [cls] token's vector. in this paper, we compare against sncse as another key baseline, not only because it is the current state-of-the-art on evaluation tasks, but also because it effectively combines contrastive learning with techniques like prompt tuning @xcite . ## conclusion we introduce hicl, the first hierarchical contrastive learning framework, highlighting the importance of incorporating local contrastive loss into the prior training paradigm. we delve into the optimal methodology for navigating the relationship between segments from same sequence in the computation of local contrastive loss. despite the extra time required for slicing sequences into segments, hicl significantly accelerates the encoding time of traditional contrastive learning models, especially for long input sequences. moreover, hicl excels in seamlessly integrating with various existing contrastive learning frameworks, enhancing their performance irrespective of their distinctive data augmentation techniques or foundational architectures. we employ simcse, esimcse, and sncse as case studies across seven sts tasks to demonstrate its scalability. notably, our implementation with the sncse backbone model achieves the new state-of-the-art performance. this makes our hierarchical contrastive learning method a promising approach for further research in this area.