W hitened CSE : Whitening-based Contrastive Learning of Sentence Embeddings (2023)

Abstract: This paper presents a whitening-based contrastive learning method for sentence embedding learning (WhitenedCSE), which combines contrastive learning with a novel shuffled group whitening. Generally, contrastive learning pulls distortions of a single sample (i.e., positive samples) close and push negative samples far away, correspondingly facilitating the alignment and uniformity in the feature space. A popular alternative to the “pushing” operation is whitening the feature space, which scatters all the samples for uniformity. Since the whitening and the contrastive learning have large redundancy w.r.t. the uniformity, they are usually used separately and do not easily work together. For the first time, this paper integrates whitening into the contrastive learning scheme and facilitates two benefits. 1) Better uniformity. We find that these two approaches are not totally redundant but actually have some complementarity due to different uniformity mechanism. 2) Better alignment. We randomly divide the feature into multiple groups along the channel axis and perform whitening independently within each group. By shuffling the group division, we derive multiple distortions of a single sample and thus increase the positive sample diversity. Consequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment. Extensive experiments on seven semantic textual similarity tasks show our method achieves consistent improvement over the contrastive learning baseline and sets new states of the art, e.g., 78.78% (+2.53% based on BERT{pasted macro ‘BA’}) Spearman correlation on STS tasks.

## introduction this paper considers self-supervised sentence representation (embedding) learning. it is a fundamental task in language processing (nlp) and can †corresponding author. 1 our code will be available at https://github.com/ supstarzh/whitenedcse . meanwhile, in (d), the positive samples after sgw (red) obtain higher diversity than the original bert features (green). using these diverse positive samples for contrastive learning, the proposed whitenedcse achieves better alignment. to improve the uniformity as well as the alignment, there are two popular approaches, i.e., contrastive learning and post-processing. 1) the contrastive learning methods @xcite @xcite pulls similar sentences close to each other and pushes dissimilar sentences far-away in the latent feature space. pulling similar sentences close directly enforces alignment, while pushing dissimilar sentences apart implicitly enforces uniformity @xcite . 2) in contrast, the postprocessing methods mainly focus on improving the uniformity. they use normalizing flows @xcite or whitening operation @xcite ) to project the already-learned representations into an isotropic space. in other words, these methods scatter all the samples into the feature space and thus improve the uniformity. in this paper, we propose a whitening-based contrastive learning method for sentence representation learning (whitenedcse). for the first time, we integrate whitening into the contrastive learning scheme and demonstrate substantial improvement. specifically, whitenedcse combines contrastive learning with a novel shuffled group whitening (sgw). given a backbone feature, sgw randomly divides the feature into multiple groups along the channel axis and perform whitening independently within each group. the whitened features are then fed into the contrastive loss for optimization. although the canonical whitening (or group whitening) is only beneficial for uniformity, sgw in whitenedcse improves not only the uniformity but also the alignment. we explain these two benefits in details as below: another important advantage of sgw is: since it is applied onto the backbone features, it incurs very slight computational overhead for generating additional positive samples. this high efficiency allows whitenedcse to increase the number of positive samples (more than common setting of 2) in a mini-batch with little cost. ablation study shows that the enlarged positive-sample number brings a further benefit. our contributions are summarized as follows: ## methods in this section, we first describe the overall architecture of whitenedcse and then present the details of all the modules, including the shuffled group whitening module and the new contrastive learning module. ## ablation analysis in this section, we further investigate the effectiveness of our proposed model whitenedcse. for all experiments, we use bert base as our base model, and evaluate whitenedcse on the sts tasks unless otherwise specified. ## conclusion in this paper, we proposed whitenedcse, a whitening-based contrastive learning framework for unsupervised sentence representation learning. we proposed a novel shuffled group whitening, which reinforces the contrastive learning effect regarding both the uniformity and the alignment. specifically, it retains the role of whitening in dispersing data, and can further improve uniformity on the basis of contrastive learning. additionally, it shuffles and groups features on channel axis, and performs whitening independently within each group. this kind of operation can be regarded as a disturbance on feature dimension. we obtain multiple positive samples through this operation, and learn the invariance to this disturbance to obtain better alignment. experimental results on seven semantic textual similarity tasks have shown that our approach achieve consistent improvement over the contrastive learning baseline.