Composition-contrastive Learning for Sentence Embeddings (2023)

Abstract: Vector representations of natural language are ubiquitous in search applications. Recently, various methods based on contrastive learning have been proposed to learn textual representations from unlabelled data; by maximizing alignment between minimally-perturbed embeddings of the same text, and encouraging a uniform distribution of embeddings across a broader corpus. Differently, we propose maximizing alignment between texts and a composition of their phrasal constituents. We consider several realizations of this objective and elaborate the impact on representations in each case. Experimental results on semantic textual similarity tasks show improvements over baselines that are comparable with state-of-the-art approaches. Moreover, this work is the first to do so without incurring costs in auxiliary training objectives or additional network parameters.

## introduction significant progress has been made on the task of learning universal sentence representations that can be used for a variety of natural language processing tasks without task-specific fine-tuning @xcite @xcite @xcite @xcite . recent works have shown the potential to learn good sentence embeddings without labeled data by fine-tuning pre-trained language models (plms) using the unsupervised framework introduced in simclr @xcite , adapted to the natural language processing (nlp) domain. in computer vision (cv), simclr exploits a series of transformations (blurs, crops, color distortions, etc.) to construct positive pairs from otherwise unique data points. a cross entropy objective @xcite is then applied to minimize distance between representations originating from the same datum, while maximizing the distance to all other points in a mini-batch. the success of the framework in computer vision is due largely to the diversity of augmentations used for creating positive pairs, which leave the identity of the original example intact while reducing pairwise mutual information in the input space @xcite @xcite . constructing positive pairs via discrete augmentations have not been effective when applying the same objective to sentence embeddings. in fact, @xcite perform an ablation study of textual augmentations (e.g., cropping, synonym replacement) and find that training on these pairs hurts downstream performance on semantic textual similarity (sts) tasks. instead, they observe that minimal (10%) dropout noise can be used to create positive pairs on-the-fly, and empirically results in stronger representations. this framework relying on nearly identical pairs is known as simcse. since the dropout noise exists as a regularization component of the bert architecture @xcite , explicit augmentations are unnecessary, making it a simple yet effective framework for unsupervised learning of sentence embeddings. here, we make a case for composition as augmentation, by exploiting its presence in language as a signal for learning sentence encoders. we conduct a series of experiments to illustrate the impact of training on positive examples derived by averaging representations of textual constituents in the latent space. following previous works, we benchmark the proposed strategy on 7 sts tasks. our results show that it is feasible to significantly improve upon simcse without making expensive architectural modifications or changing the overall training objective. we hope our findings can inspire new avenues of inquiry in text representation learning that draw on long-standing notions in semantics and linguistics. in a bustling city, the young artist creates her masterpiece under the moonlight. in a bustling city, the young artist creates her masterpiece under the moonlight. 2 background and related work ## composition-based contrastive learning our augmentation strategy retains the simplicity and efficiency of simcse, as illustrated in fig-ure 1. specifically, it requires just one additional forward pass that is ultimately compensated by a non-trivial reduction in convergence time ( §6). beginning with a corpus of unlabelled sentences {x i } m @xmath0, we consider x + i only in the latent space, as a composition of the representations of (x we experiment with aggregation methods in §5, and find that the best approach varies according to the size and type of underlying plm. in our final model based on bert base , we find that this manner of augmentation is especially suitable for the scheme proposed in directclr @xcite , which aims to directly mitigate dimensional collapse by computing the loss from eq. 1 on a subset of the embedding vector axes before backpropagating to the entire representation. decomposition as data augmentation. to explain the motivation for decomposing examples in the input space, we can consider an example from the development subset of sts-b labelled as having high semantic similarity: a man is lifting weights in a garage. a man is lifting weights. there are two semantic atoms at play in the first text: 1) a man is lifting weights, and 2) a man is in a garage. the similarity between the two texts can only be considered high based on the first atom; lifting weights. it cannot be said that there is a general relation between being in a garage and lifting weights -a garage is equally, if not more likely to be related to cars, parking, or storage, yet this does not preclude a connection between them. it is only through the composition of both atoms that we can relate the two. thus, there is a need for sentence encoders to learn more generalized phrase representations; to at least implicitly abide by principles of semantic compositionality. the challenge in enforcing this kind of constraint through a contrastive objective is in the choice of data -it would require a corpus where lexical collocations are encountered across a diverse set of contexts. subsampling from decomposed inputs. to further examine the effect of decomposition in the input space, we leverage a pre-trained discourse parser 2 to extract atomic semantic units from each unique example in the training set; typically simple phrases or clauses. we experiment with 3 kinds of strategies (figure 1a ) to expand the training set, besides considering our augmentation in isolation: let @xmath1,k } c @xmath2: ## ablation we ablate several aspects of the approach to understand their impact in isolation. we first consider the subsampling strategy, or lack thereof, in which each model achieves the best sts-b development set performance. these are then tied to each model in subsequent ablations. including subsamples. aggregration method. in sbert (reimers and gurevych, 2019), and in historically effective works such as infersent @xcite , plms are fine-tuned with a cross entropy loss to predict whether two sentences u and v entail or contradict eachother. their pooler is a concatenation of the two sentence embeddings, along with second-order features such as the element-wise difference, |u -v|. we experiment with these aggregration methods, as well as simpler choices such as element-wise sums/averages. we can see in ## analysis stability and efficiency of training. successors to simcse have incrementally improved sts performance while disproportionately driving up resource requirements. this limits accessibility to practitioners who wish to learn embeddings from their own corpora, perhaps in other languages. differently, our approach relies on a single additional forward pass while converging much faster than simcse. in figure 3 , we compare our bert base model's evaluation curve to simcse's for 1000 training steps in the same setting. we observe that composition as augmentation greatly speeds up convergence, with evaluation metrics plateauing much faster, and more stably than simcse. in fact, on a single nvidia a100 gpu (40gb), our model can finish training in under 15 minutes. text length as a feature. to investigate the structure of the learned space, in figure 5 , we visualize embeddings of sentences from the development set of sts-b after down-projecting to 2d euclidean space. we employ umap @xcite with cosine distance as the metric to preserve local and global topological neighborhoods. the same parameters are used to compute the embeddings in figure 5a and 5b, which are derived from dropout noise, and composition-based augmentations (w/ composition) respectively. in figure 5a , we can observe several clusters of dark points corresponding to shorter sentences. this corroborates our intuition that minimal augmentation to create positive pairs can lead to shortcut learning, wherein text length is relied upon to solve the training objective. in contrast, we see a more scattered distribution of points in figure 5b , particularly with shorter sen-tences. coupled with the improved performance on sts tasks, we can conclude that our framework is less prone to learning from spurious correlations. ## conclusion in summary, we proposed a new way to construct positive pairs for unsupervised contrastive learning frameworks relying on pre-trained language models. our experiments on sts tasks verified the effectiveness of the approach, which achieved competitive results with more complex learning methods, with the benefit of stabilizing and reducing the overall cost of training. we provided empirical studies and qualitative examinations into our approach, verifying its ability to train sentence encoders with better alignment. we believe this work can foster new avenues of inquiry in contrastive learning, especially those that draw upon a human cognition of language.