S um CSE : Summary as a transformation for Contrastive Learning (2024)

Abstract: Sentence embedding models are typically trained using contrastive learning (CL), either using human annotations directly or by repurposing other annotated datasets. In this work, we explore the recently introduced paradigm of generating CL data using generative language models (LM). In CL for computer vision (CV), compositional transformations (series of operations applied over an image. e.g. cropping + color distortion) which modify the input/image to retain minimal information were shown to be very effective. We show that composition of a ‘Summary’ transformation with diverse paraphrasing/contradicting transformations accomplishes the same and works very well in CL for sentence embeddings. Our final generated dataset (using Vicuna-13B) significantly outperforms the previous best unsupervised method (using ChatGPT) by 1.8 points, and SimCSE, a strong supervised baseline by 0.3 points on the semantic text similarity (STS) benchmark.

## introduction contrastive learning (cl) is widely used for training sentence embeddings @xcite @xcite . cl typically uses data in the form of (anchor, positive, negative) where embedding of anchor is made closer to positive and away from the negative. simcse @xcite and promptbert @xcite used manually annotated nli data (275k) to train sentence similarity models. reliance on annotated data, however, limits contrastive learning from being performed at scale and from being transferred to other domains. some works repurposed existing web datasets to be used as contrastive learning data - @xcite used super-ni @xcite ; @xcite -summer internship work. academic collaborators created the dataset. meta was not involved in using the dataset for training. meta did not perform the experiments or use or process any of the data described or referenced in the paper. on the other hand, in cl for computer vision, @xcite ; @xcite showed the effectiveness of compositional transformations that retain minimum information necessary for downstream tasks. we draw inspiration from these works to propose composition of a 'summary' transformation over other diverse 'paraphrase' and 'contradiction' transformations, as positives and negatives respectively, in cl for sentence embeddings. synthetic dataset generated by our unsupervised methodology, sumcse (uses vicuna-13b), improves over syncse (uses chatgpt) (+1.8) and simcse (+0.3), on sts. our method shows benefits (+0.9) when directly applied to other existing datasets and achieves an additional (+0.5) over sumcse when scaled (4x synthetic dataset), on sts. ## background and motivation the success of a contrastive algorithm is substantially influenced by these transformations @xcite . minimising infonce loss learns representations that maximise the lower bound on mutual information between input and transformation, i(h, h + ), over the dataset. @xcite used this to theoretically justify the 'infomin' principle -an ideal transformation should share with the input minimal information necessary to perform well at a downstream task. it should minimise all irrelevant nuisances in the input. also, simclr @xcite showed that compositional transformations work very well for cl training image representations and found that (image_cropping + colour_distortion) worked the best. cropping follows the infomin principle (reduces mutual information with the input). it matches global-local contexts in an image to learn robust representations. inspired by these concepts, we propose transformations for sentence cl. ## methodology our goal here is unsupervised generation of positives and negatives given an anchor. the generated synthetic dataset is used to train an embedding model with eqs. 1/ 2. this trained model is used to evaluate results on sts and other benchmarks.