O ss CSE : Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding (2023)

Abstract: Contrastive learning has been demonstrated effective in unsupervised sentence representation learning. Given one sentence, positive pairs are obtained by passing the sentence to the encoder twice using the different dropout masks, and negative pairs are obtained by taking another sentence in the same mini-batch. However, the method suffers from the surface structure bias, i.e., sentences with similar surface structures will be regarded as close in semantics while sentences with dissimilar surface structures will be viewed as distinct in semantics. This leads to the result that paraphrasing a sentence that is dissimilar in surface structure will receive a lower semantic similarity score than inserting a negative word into the sentence. In this paper, we first verify the bias by collecting a sentence transformation testset. Then we systematically probe the existing models by proposing novel splits based on benchmark datasets in accordance with semantic and surface structure similarity. We tackle the bias in two aspects: balancing the learning target by augmenting with data that counters the bias, and meanwhile preserving word semantics by leveraging recall loss to prevent catastrophic forgetting. We evaluate our model on standard semantic textual similarity (STS) tasks using different pre-trained backbones and achieve state-of-the-art averaged performance across the STS benchmarks. Particularly, our models that are fine-tuned with RoBERTa_{base} and RoBERTa_{large} achieve significantly better performance on most benchmark datasets.

## introduction deep and surface structures @xcite @xcite for a linguistic expression, e.g., a sentence, are often used as simple binary opposite terms, with the deep structure * work done while interning at amazon representing the semantics, and the surface structure being the actual sentence we see. the deep structure of a linguistic expression is a construct seeking to unify several related surface structures. a particular deep structure can be represented by multiple surface structures, e.g., "i purchased some beautiful clothes" and "some beautiful clothes were bought by me" are two surface structures that can be unified by the same deep structure. recent studies @xcite show that contrastive learning schemes @xcite that are fine-tuned on deep transformer-based language models pretrained on large generic corpora, e.g., @xcite and roberta @xcite , have significantly benefited the sentence representation learning. the idea of contrastive learning is that positive and negative pairs are constructed given a mini-batch of sentences in training. specifically, for unsupervised sentence representation learning, the positive pair is composed of two identical sentences and the negative pair is constructed by one sentence with another sentence in the minibatch. however, this method for forming positive and negative pairs may lead to an inherent bias that the sentences with almost, if not completely, identical surface structures foot_0 have the same semantic meaning while the sentences with dissimilar surface structures have distinct semantic meanings, leading to the result that semantic meanings are consistently biased towards with surface structure similarities. the bias presents two intermediate questions: @xcite to what extent do current models suffer the effects of bias? (2) how to boost the model performance by overcoming the bias? to answer the first question, we first propose to validate whether current models can correctly rank ss sim ds sim low high low cont. oppn. high oppn. cont. a few sentence transformations. furthermore, to systematically evaluate how the bias effect existing models, we split the existing datasets following the consistency (cont.) and opposition (oppn.) settings by surface structure and deep structure similarity in to answer the second question, since the major bottleneck of current models is their poor performance on the oppn. datasets, we apply two strategies: (1) automatic sentence-level data augmentations in accordance with the oppn. setting with max margin loss applied on the augmented data; (2) a regularization loss to prevent catastrophic forgetting in token-level semantics which is critical to constitute sentence meanings. in this work, we use match error rate (mer) @xcite to measure the surface structure similarity, which is based on the edit distance @xcite . edit distance is often used for quantitatively capturing the linguistic characteristics of paraphrase pairs @xcite @xcite . in short, we make the following contributions: ## related work unsupervised sentence representation learning has been widely studied. the early study attempts to leverage sentence internal structure @xcite @xcite , and sentence context @xcite to learn the representation. @xcite proposes sent2vec, a simple unsupervised method that obtains sentence embeddings by combining word vectors with n-gram embeddings. several efforts have been made to obtain sentence representations using the embedding of some special token or averagely pooled last layer representations, due to the great success of pre-trained language models @xcite . however, @xcite identified the anisotropy problem that the native embeddings from plms are concentrated in a small cone in the vector space. to address this issue, techniques such as bert-flow @xcite and bert-whitening @xcite have been proposed as post-processing methods. recently, unsupervised sentence embeddings have utilized contrastive learning schemes to further boost the performance by different data augmentation methods, such as dropout @xcite , noise-based negatives @xcite , deletion-reordering-substitution @xcite , adversarial attack @xcite , span sampling @xcite , entity augmentations @xcite and bert layer sampling @xcite . in addition to the above research that create pairs of positive and negative representations in different ways, there have been research focusing on improved learning objectives, @xcite al. (2021b) of template denoising in the contrastive loss. although achieving promising results, these methods didn't consider reducing the surface structure bias for contrastive learning in their data augmentation methods or modified learning objectives. ## probing 3.1 sentence transformations to specifically exhibit the bias, we write down seven differently transformed sentences as shown in example 1 considering six types of sentence transformations: ( 1 we probe the officially released unsupervised and supervised models with bert-base-uncased as the backbone from simcse @xcite 2 . each sentence has three associated values: simi-2 https://github.com/princeton-nlp/simcse larity values between the original and transformed sentence calculated by both unsupervised and supervised models, and an mer value that measures surface structure similarity. the mer value (range from 0 ∼ 1) indicates the proportion of words that were incorrectly deleted, substituted and inserted: the lower the value, the higher the surface structure similarity. ## method unsupervised contrastive learning given a dataset of n paired sentences {s i , s + i } n @xmath0, where s i and s + i are semantically similar and regarded as a positive pair. the main idea behind unsupervised contrastive learning is to utilize identical sentences to construct positive pairs. i.e., s + @xmath1, and random two sentences in a mini-batch as negative pairs. we feed s i to the encoder twice by applying different dropout masks in each forward pass and obtaining two sentence embeddings h i , h + i as shown in figure 3 . with h i and h + i for each sentence in a mini-batch with batch size n , the contrastive learning objective is formulated as below: step 1: generating a semantically contradicted example sentence s n i by only inserting negative words to the original sentence s i , e.g., my dog likes eating @xmath2. given the sentence, we use dependency parsing to find the position of the word that is tagged as root as shown in figure 4 . then we insert the negative word based on the pos tag of the root word, e.g., if its pos tag is vbz, then we change "likes" → "does not like". step 2: given a sentence s i , we aim to generate a sentence s p i that is similar in semantics but distinctive in surface structure. we use back translation models on english-russian-english, english-german-english foot_2 to obtain such candidate sentences. to filter the desired back-translation sentences that satisfy the requirement of surface structure similarity, we then select the examples of which their mer meets the requirement (refer to appendix a for details) hence, we propose the max margin loss based on the h p i and h n i representations from s p i and s n i : although h n i is semantically contradictory to the original sentence, it is close to the original one in terms of semantic relatedness. hence, we design the above two-way max marginal loss, which can guide the model to learn the following objective: α < sim(h i , h p i )sim(h i , h n i ) < β where α, β are margins. catastrophic forgetting as shown in figure 6 , we also observe that the training will suffer from performance degradation on oppo. dataset splits after 3,000 training steps (0.3 epoch). it might be due to the catastrophic forgetting of word semantics and oppn. splits are more relied on word semantics rather than surface structure. however, the fine-tuning process with pre-trained language models is prone to catastrophic forgetting, and it will cause a significant drop in performance on the oppn. dataset splits particularly. we use the learning objective from @xcite , which proposes a pretraining simulation mechanism to recall the knowledge from pretraining tasks d s without data. ## conclusion we investigate the surface structure bias in contrastive learning for the unsupervised sentence embedding and systematically probe the bias by constructing datasets following the two designated settings on the surface and deep structure similarity. we overcome the bias by data augmentation methods and then use the max margin loss to incorporate these data in the contrastive learning framework. we also use a recall loss to reduce catastrophic forgetting in unsupervised learning to preserve the word semantics in the pre-trained models. the results significantly outperform the baselines and achieve state-of-the-art results on averaged performance with different pre-trained backbones.