mi CSE : Mutual Information Contrastive Learning for Low-shot Sentence Embeddings (2023)

Abstract: This paper presents miCSE, a mutual information-based contrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the structural consistency across augmented views for every sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to state-of-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding.

## introduction measuring sentence similarity has been challenging due to the ambiguity and variability of linguistic expressions. the community's strong interest in the topic can be attributed to its applicability in numerous language processing applications, such as sentiment analysis, information retrieval, and semantic search @xcite . language models perform well on these tasks but typically require fine-tuning on the downstream task and corpora @xcite @xcite . in terms of sentence embeddings, contrastive learning schemes have already been adopted successfully @xcite @xcite . the idea of contrastive learning is that positive and negative pairs are generated given a batch of samples. whereas the positive pairs are obtained via augmentation, negative pairs are often created by random collation of sentences. following the construction of pairs, contrastive learning forces the network to learn feature representations by pushing apart different samples (negative pairs) or pulling together similar ones (positive pairs). while some methods seek to optimize for selecting "hard" negative for negative pair generation @xcite , others investigated better augmentation techniques for positive pair creation. in this regard, many methods have been proposed to create augmentations to boost representation learning. standard approaches for the augmentation aim at input data level (a.k.a discrete augmentation), which comprises word level operations such as swapping, insertion, deletion, and substitution @xcite @xcite . in contrast to that, continuous augmentation operates at the representation level, comprising approaches like interpolation or "mixup" on the embedding space @xcite @xcite . most recently, augmentation was also proposed in a more continuous fashion operating in a parameter level via simple techniques such as drop-out @xcite @xcite or random span masking @xcite . the intuition is that "drop-out" acts as minimal data augmentation, providing an expressive semantic variation. however, it will likely affect structural alignment across views. since positive pairs are constructed from identical sentences, we hypothesize that the structural dependency over the views should be preserved by utilizing drop-out noise. building on this idea, we maximize the structural dependence by enforcing distributional similarity over the attention values across the augmentation views. to this end, we employ maximization of the mutual information (mi) on the attention tensors of the positive pairs. however, since attention tensors can be very high-dimensional, computing mi can quickly become a significant burden if not intractable. this paper proposes a simple solution to alleviate the computational burden of mi computation, which can be deployed efficiently. similar to @xcite , we adopt the log-normal distribution to model attention. empirical evidence confirms this model as a good fit while facilitating the optimization objective to be defined in closed form. in this case, mutual information can be provably reformulated as a function of correlation, allowing native gpu implementation. as discussed above, the proposed approach builds upon the contrastive learning paradigm known to suffer from model collapse. this issue becomes even more problematic when enforcing mi on the attention level, as it tightens the positive pairs via regularizing the attention. therefore the selection of negative pairs becomes more critical in our setup. to this end, we utilize momentum contrastive learning to generate harder negatives @xcite . a "tighter" binding on positive pairs and repulsion on "harder" negative pairs empowers the proposed contrastive objective, yielding more powerful representations. combining ideas from momentum contrastive learning and attention regularization, we propose micse, a conceptually simple yet empirically powerful method for sentence embedding, with the goal of integrating semantic and structural information of a sentence in an information-theoretic and transformer-specific manner. we conjecture the relation between attention maps and a form of syntax to be the main driver behind the success of our approach. we speculate that our proposed method injects structural information into the model as an inductive bias, facilitating representation learning with fewer samples. the adopted structural inductive biases provide a "syntactic" prior as an implicit form of supervision during training @xcite , which promotes few-shot learning capabilities in neural language models. to validate this, we introduced a low-shot setup for training sentence embeddings. in this benchmark, we finetune the language model only with a small number of training samples. note that this is a very challenging setup. the inherent difficulty can be attributed to the need to mitigate the domain shift in the low-shot self-supervised learning scheme. we emphasize the importance of this task, as in many real-world applications, only small datasets are often available. such cases include nlp for low-resource languages or expert-produced texts (e.g., medical records by doctors), personalized lm for social media analysis (e.g., personalized hate speed recognition on twitter), etc. our proposed method significantly improves over the state-of-theart in the low-shot sentence embedding benchmark. this is the first work that explores how to combine semantic and structural information through attention regularization and empirically demonstrates this benefit for low-shot sentence embeddings. previous works: recently, vascl @xcite , consert @xcite , pcl @xcite and @xcite proposed contrastive representation learning with diverse augmentation strategies on positive pair. however, we proposed a principled approach for enforcing alignment in positive pairs at contrastive learning without discretely augmenting the data. similar to us, esimcse @xcite and mocose @xcite proposed to exploit a momentum contrastive learning model with negative sample queue for sentence embedding to boost uniformity of the representations. however, unlike us, they do not enforce any further tightening objective on the positive pairs nor consider few-shot learning. very recently, authors in informin-cl @xcite and infocse @xcite proposed information minimization-based contrastive learning. specifically, the authors propose to minimize the information entropy between positive embeddings generated by drop-out augmentation. our model differs from this paper and the method in @xcite @xcite @xcite , which focuses on using mutual information for selfsupervised learning. a key difference compared to these methods is that they estimate mi directly on the representation space. in contrast, our method computes the mi on attention. other related work include @xcite @xcite . the contributions of the proposed work are: first, we propose to inject structural information into language models by adding an attention-level objective. second, we introduce attention mutual information (ami), a sample-efficient selfsupervised contrastive learning. third, we introduce low-shot learning for sentence embedding. we show that our method performs comparably to the state-of-the-art in the full-shot scenario and significantly better in few-shot learning. ## method the proposed approach aims to exploit the structure of the sentences in a contrastive learning scheme. compared to conventional contrastive learning that solely operates at the level of semantic similarity in the embedding space, the proposed approach injects structural information into the model. this is achieved by regularizing the attention space of the model during training. we let d denote a dataset consisting of string sequences (sentences) from corpus x with @xmath0, x 2 , ..., x |x | }, where we assume x i to be a tokenized sequence of length n with x @xmath1. for mapping the input data to the embedding space, we use a bi-encoder f θ parametrized by θ. bi-encoders entail the computation of embeddings for similarity comparison, whereby each sentence in a pair is encoded separately. hence, the instantiation of a bi-encoder on augmented input data induces multiple views. for the following, we let @xmath2, 2} denote the index of the view, where each view corresponds to a different augmentation. consequently, encoding a data batch d b yields embedding matrices e @xmath3, where u denotes the dimensionality of the embeddings. employing a transformer, encoding the input data yields the embedding matrices and the associated attention tensors w v . then learning representation of the proposed approach entails the optimization of a joint loss: here, l c is responsible for the semantic alignment, corresponding to the standard infonce @xcite loss that seeks to pull positive pairs close together while pushing away negative pairs in the embedding space. in contrast, l d is responsible for the syntactic alignment, operating on the attention space. however, in comparison to l d is employed only on positive pairs' attention tensors. ## experiments in this section, we describe the experimental setting used for the evaluation, present our main results, and discuss different aspects of our method by providing several empirical analyses. ## conclusion we proposed a method to inject structural similarity into language models for self-supervised representation learning for sentence embeddings. the proposed approach integrates the inductive bias at the level of transformer attention by enforcing mutual information on positive pairs obtained by drop-out augmentation. leveraging attention regularization makes the proposed approach much more sample efficient. consequently, it outperforms methods with a significant margin in low-shot learning scenarios while having state-of-the-art performance in full-shot to comparable approaches. kun zhou, beichen zhang, wayne xin zhao, and ji-rong wen. 2022a. debiased contrastive learning of unsupervised sentence representations. arxiv preprint arxiv:2205.00656. kun zhou, beichen zhang, xin zhao, and ji-rong wen. 2022b. debiased contrastive learning of unsupervised sentence representations. in acl, pages 6120-6130.