A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings (2022)

Abstract: Contrastive learning has shown great potential in unsupervised sentence embedding tasks, e.g., SimCSE (CITATION).However, these existing solutions are heavily affected by superficial features like the length of sentences or syntactic structures. In this paper, we propose a semantic-aware contrastive learning framework for sentence embeddings, termed Pseudo-Token BERT (PT-BERT), which is able to explore the pseudo-token space (i.e., latent semantic space) representation of a sentence while eliminating the impact of superficial features such as sentence length and syntax. Specifically, we introduce an additional pseudo token embedding layer independent of the BERT encoder to map each sentence into a sequence of pseudo tokens in a fixed length. Leveraging these pseudo sequences, we are able to construct same-length positive and negative pairs based on the attention mechanism to perform contrastive learning. In addition, we utilize both the gradient-updating and momentum-updating encoders to encode instances while dynamically maintaining an additional queue to store the representation of sentence embeddings, enhancing the encoder’s learning performance for negative examples. Experiments show that our model outperforms the state-of-the-art baselines on six standard semantic textual similarity (STS) tasks. Furthermore, experiments on alignments and uniformity losses, as well as hard examples with different sentence lengths and syntax, consistently verify the effectiveness of our method.

## introduction sentence embedding serves as an essential technique in a wide range of applications, including semantic search, text clustering, text classification, etc. @xcite @xcite @xcite . contrastive figure 1 : a realistic scenario is described at the top, negative examples have the same length and structure, while positive examples act in the opposite way. in comparison, discrete augmentation obtains positive instances with word deletion or reordering @xcite , which may misinterpret the meaning. the continuous method treats embeddings of the same original sentence as positive examples and augments sentences with the different encoding functions @xcite . learning works on learning representations such that similar examples stay close whereas dissimilar ones are far apart, and thus is suitable for sentence embeddings due to its natural availability of similar examples. incorporating contrastive learning in sentence embeddings improves the efficiency of semantic information learning in an unsupervised manner @xcite and has been shown to be effective on a variety of tasks @xcite @xcite . in contrastive learning for sentence embeddings, a key challenge is constructing positive instances. both discrete and continuous augmentation methods have been studied recently. methods in @xcite ; @xcite perform discrete operations directly on the original sentences, such as word deletion and sentence shuffling, to get positive samples. however, these methods may lead to unacceptable semantic distortions or even complete misinterpretations of the original statement. in contrast, the simcse method @xcite obtains two different embeddings in the continuous embedding space as a positive pair for one sentence through different dropout masks @xcite in the neural network for representation learning. nonetheless, this method overly relies on superficial features existing in the dataset like sentence lengths and syntactic structures and may pay less reflection on meaningful semantic information. as an illustrative example, the sentencepair in fig. 1 "a caterpillar was caught by me." and "i caught a caterpillar." appear to organize differently in expression but convey exactly the same semantics. to overcome these drawbacks, in this paper, we propose a semantic-aware contrastive learning framework for sentence embeddings, termed pseudo-token bert (pt-bert), that is able to capture the pseudo-token space (i.e., latent semantic space) representation while ignoring effects of superficial features like sentence lengths and syntactic structures. inspired by previous works on prompt learning and sentence selection @xcite @xcite , which create a pseudo-sequence and have it serve the downstream tasks, we present pt-bert to train pseudo token representations and then to map sentences into pseudo token spaces based on an attention mechanism. in particular, we train additional 128 pseudo token embeddings, together with sentence embeddings extracted from the bert model (i.e., gradient-encoder), and then use the attention mechanism @xcite to map the sentence embedding to the pseudo token space (i.e., semantic space). we use another bert model (i.e., momentum-encoder @xcite ) to encode the original sentence, adopt a similar attention mechanism with the pseudo token embeddings, and finally output a continuously augmented version of the sentence embedding. we treat the representations of the original sentence encoded by the gradientencoder and the momentum-encoder as a positive pair. in addition, the momentum-encoder also generates negative examples, dynamically maintains a queue to store these negative examples, and updates them over time. by projecting all sentences onto the same pseudo sentence, the model greatly reduces the dependence on sentence length and syntax when making judgments and makes the model more focused on the semantic level information. in our experiments, we compare our results with the previous state-of-the-art work. we train pt-bert on 10 6 randomly sampled sentences from english wikipedia and evaluate on seven standard semantic textual similarity (sts) tasks @xcite @xcite @xcite . besides, we also compare our approach with a framework based on an advanced discrete augmentation we proposed. we obtain a new stateof-the-art on standard semantic textual similarity tasks with our pt-bert, which achieves 77.74% of spearman's correlation. to show the effectiveness of pseudo tokens, we calculate the align-loss and uniformity loss @xcite ) and verify our approach on a sub-dataset with hard examples sampled from @xcite . we have released our source code foot_0 to facilitate future work. ## related work in this section, we discuss related studies with repect to the contrastive learning framework and sentence embedding. ## methods in this section, we introduce pt-bert, which provides novel contributions on combining advantages of both discrete and continuous augmentations to advance the state-of-art of sentence embeddings. we first present the setup of problems with a thorough analysis on the bias introduced by the textual similarity theoretically and experimentally. then we show the details of pseudo-token representation and our model's architecture. beddings @xmath0. with a slight change of the encoding function (e.g., encoders with different dropout masks), h ′ can be seen as a more precisely augmented version of h compared with the discrete augmentation. semantic information of h ′ should be the same as h. therefore, h and h ′ are a pair of positive examples and we could randomly sample a sentence to construct negative example pairs. ## experiments in this section, we perform the standard semantic textual similarity (sts) @xcite @xcite @xcite tasks to test our model. for all tasks, we measure the spearman's correlation to compare our performance with the previous stateof-the-art simcse @xcite . in the following, we will describe the training procedure in detail. ## conclusion in this paper, we propose a semantic-aware contrastive learning framework for sentence embeddings, termed pt-bert. our proposed pt-bert approach is able to weaken textual similarity information, such as sentence length and syntactic structures, by mapping the original sentence to a fixed pseudo sentence embedding. we provide analysis of these factors on methods based on continuous and discrete augmentation, showing that pt-bert augments sentences more accurately than discrete methods while considering more semantics instead of textual similarity than continuous approaches. lower uniformity loss and alignment loss prove the effectiveness of pt-bert and further experiments also show that pt-bert could handle hard examples better than existing approaches. providing a new perspective to the continuous data augmentation in sentence embeddings, we believe our proposed pt-bert has great potential to be applied in broader downstream applications, such as text classification, text clustering, and sentiment analysis.