An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learning (2022)

Abstract: Unsupervised sentence embeddings learning has been recently dominated by contrastive learning methods (e.g., SimCSE), which keep positive pairs similar and push negative pairs apart. The contrast operation aims to keep as much information as possible by maximizing the mutual information between positive instances, which leads to redundant information in sentence embedding. To address this problem, we present an information minimization based contrastive learning InforMin-CL model to retain the useful information and discard the redundant information by maximizing the mutual information and minimizing the information entropy between positive instances meanwhile for unsupervised sentence representation learning. Specifically, we find that information minimization can be achieved by simple contrast and reconstruction objectives. The reconstruction operation reconstitutes the positive instance via the other positive instance to minimize the information entropy between positive instances. We evaluate our model on fourteen downstream tasks, including both supervised and unsupervised (semantic textual similarity) tasks. Extensive experimental results show that our InforMin-CL obtains a state-of-the-art performance.

## introduction how to learn universal sentence embeddings by large-scale pre-trained models @xcite , such as bert, has been studied extensively in the literature @xcite . recently, contrastive learning has been used widely to learn better sentence embeddings @xcite . generally, contrastive learning uses various data augmentation methods to generate different views of the input sentences and samples positive instances and ## method we propose an informin-cl model for unsupervised sentence representation learning (figure 1 ). there are two main steps, contrast and reconstruction. we first present the contrast objective to learn the useful information: we push apart posi-tive instances and negative instances while pulling positive instances together, which implies maximizing mutual information between positive instances. later we present the reconstruction task to drop the useless information: we minimize the conditional information entropy of one positive instance given the other positive instance. algorithm 1 provides the pseudo-code of informin-cl. algorithm 1 pseudocode of informin-cl in a pytorch-like style. input: batch size n, temperature τ , structure of f and γ. ## limitations although our method outperforms baselines on both unsupervised and supervised tasks in most cases, there are still at least two limitations. first, we simply sample negative instances from other sentences in the mini-batch, which may lead to false negatives. punishing false negatives during training by assigning lower weight for negatives with higher similarity may be a solution. second, although redundant information is discarded, what redundant information forgets and remains is unknown. it would be interesting to explore this problem by integrating interpretation methods. ## conclusion in this work, we propose informin-cl, an effective contrastive learning approach, which improves state-of-the-art sentence embedding performance on downstream tasks. informin-cl discards redundant information stored in positive instances by encouraging one positive instance to reconstruct the other positive instance in the same pair. we test informin-cl on seven supervised and seven unsupervised tasks. experimental results indicate our method outperforms all previous competitors.