Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words (2023)

Abstract: The performance of sentence encoders can be significantly improved through the simple practice of fine-tuning using contrastive loss. A natural question arises: what characteristics do models acquire during contrastive learning? This paper theoretically and experimentally shows that contrastive-based sentence encoders implicitly weight words based on information-theoretic quantities; that is, more informative words receive greater weight, while others receive less. The theory states that, in the lower bound of the optimal value of the contrastive learning objective, the norm of word embedding reflects the information gain associated with the distribution of surrounding words. We also conduct comprehensive experiments using various models, multiple datasets, two methods to measure the implicit weighting of models (Integrated Gradients and SHAP), and two information-theoretic quantities (information gain and self-information). The results provide empirical evidence that contrastive fine-tuning emphasizes informative words.

## introduction embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing (nlp) @xcite @xcite . such sentence embedding methods can also embed text of various types and lengths, such as queries, passages, and paragraphs; therefore, they are widely used in diverse applications such as information retrieval @xcite , question answering @xcite , and retrieval-augmented generation @xcite . one of the earliest successful sentence embedding methods is additive composition @xcite , which embeds a sentence (i.e., a sequence of words) by summing its static word embeddings (swes; @xcite . besides, weighing each word based on the inverse of word frequency considerably improved the quality of the sentence embeddings, exemplified by tf-idf @xcite and smoothed inverse word frequency (sif; @xcite . recent sentence embeddings are built on masked language models (mlms; @xcite @xcite . although sentence embeddings from the additive composition of mlms' word embeddings are inferior to those of swes @xcite , fine-tuning mlms with contrastive learning objectives has elevated the quality of sentence embeddings @xcite @xcite and is now the de-facto standard. interestingly, these contrastive-based sentence encoders do not employ explicit word weighting, which is the key in the swe-based methods. in this paper, we demonstrate that a reason for the success of contrastive-based sentence encoders is implicit word weighting. specifically, by com-bining explainable ai (xai) techniques and information theory, we demonstrate that contrastivebased sentence encoders implicitly weight each word according to two information-theoretic quantities (figure 1 ). to measure the contribution (i.e., implicit weighting) of each input word to the output sentence embedding within the encoders, we used two xai techniques: integrated gradients (ig; @xcite and shapley additive explanations (shap; lundberg and lee, 2017) (section 3.1). to measure the information-theoretic quantities of each word, we used the two simplest quantities, information-gain kl(w) and selfinformationlog p (w) (section 3.2). to demonstrate our hypothesis, we first provided a theoretical connection between contrastive learning and information gain (section 4). we then conducted comprehensive experiments with a total of 12 models and 4 datasets, which found a strong empirical correlation between the encoders' implicit word weighting and the information-theoretic quantities (section 5). the results of our study provide a bridge between swe-era explicit word weighting techniques and the implicit word weighting used by recent contrastive-based sentence encoders. ## contrastive-based sentence encoders this section provides an overview of contrastivebased sentence encoders such as sbert @xcite foot_0 and simcse @xcite . these models are built by fine-tuning mlms with contrastive learning objectives. input and output: as shown in figure 1 , a contrastive-based sentence encoder m : r n×@xmath0, . . . , w n ] ∈ r n×d corresponding to input words and special tokens. for example, for the sentence "i like playing the piano," the input to the encoder m is @xmath1, w i , w like , . . . , w @xcite ], and sentence embedding s is calculated as follows: architecture: the encoder architecture consists of (i) transformer layers (same as mlm architecture) and (ii) a pooling layer. (i) transformer layers update input word embeddings to contextualized word embeddings w @xmath2. (ii) the pooling layer then pools the n representations [e 1 , . . . , e n ] into a single sentence embedding @xmath3. there are two major pooling methods, mean and cls: mean averages the contextualized word embeddings, and cls just uses the embeddings for a [cls] token after applying an mlp on top of it. contrastive fine-tuning: the contrastive finetuning of mlms is briefly described below. in contrastive learning, positive pairs (s, s pos ), i.e., semantically similar pairs of sentence embeddings, are brought closer, while negative pairs (s, s neg ) are pushed apart in the embedding space. positive examples s pos and negative examples s neg for a sentence embedding s are created in different ways depending on the method. for instance, in the unsupervised simcse @xcite , a positive example s pos is created by embedding the same sentence as s by applying a different dropout; and a negative example s neg is created by embedding a sentence randomly sampled from a training corpus. ## analysis method we compare the implicit word weighting within the contrastive-based sentence encoders with information-theoretic quantities of words. here we introduce (i) quantification of the implicit word weighting within the encoders using two xai techniques (section 3.1) and (ii) two informationtheoretic quantities of words (section 3.2). ## theoretical analysiss this section provides brief explanations of the theoretical relationship between kl(w) and contrastive learning. given a pair of sentences (s, s ′ ), contrastive learning can be regarded as a problem of discriminating whether a sentence s ′ is a positive (semantically similar) or negative (not similar) example of another sentence s. we reframe this discrimination problem using word frequency distribution. after observing w in s, the positive example is likely to contain words that co-occur with w; i.e., the word distribution of the positive example likely follows p sent (•|w). contrary, the negative example is likely to contain random words from a corpus regardless of the observation of w; i.e., the word distribution of the negative example likely follows p (•). hence, kl(p sent (•|w) ∥ (p (•)) = kl(w) approximately represents the objective of the discrimination problem (i.e., contrastive learning). see appendix c for a more formal explanation. ## experiments in this section, we investigate the empirical relationship between the implicit word weighting within contrastive-based sentence encoders (quantified by ig or shap) and the information-theoretic quantities (kl(w) or log p (w)). ## conclusion we showed that contrastive learning-based sentence encoders implicitly weight informative words based on information-theoretic quantities. this indicates that the recent sentence encoders learn implicit weightings similar to the explicit ones used in the swe-based methods. we also provided the theoretical proof that contrastive learning induces models to weight each word by kl(w). these provide insights into why contrastive-based sentence encoders succeed in a wide range of tasks, such as information retrieval @xcite and question answering @xcite , where emphasizing some informative words is effective. besides sentence encoders, investigating the word weighting of retrieval models is an interesting future direction.