Normalized Contrastive Learning for Text-Video Retrieval (2022)

Abstract: Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contrastive learning suffers from incorrect normalization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, significantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Empirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state-of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering.

## introduction with the advent of large-scale multimodal data and transformer-based architectures, cross-modal contrastive learning has contributed to the recent advances in multimodal representation learning @xcite . cross-modal contrastive learning provides a simple, yet highly effective approach for learning representations from multimodal data without supervisions. in particular, clip (contrastive language-image pretraining) @xcite learns image-text representations using vision and text transformer encoders on millions of image-text data from the web and demonstrates that the learned vision and text encoders can perform zero-shot transfer on various vision tasks. more recently, clip4clip (luo et al., * work done during internship at meta 2021) extends the pretrained clip model and finetunes it on text-video datasets for embedding-based text-video retrieval, achieving state-of-the-art performance. in embedding-based retrieval, the retrieval probabilities of each text or video instance are defined by their embedding similarity to their cross-modal queries. when there is one-to-one correspondence between text and video instances as in many standard benchmarks, one would expect that the retrieval probabilities of a video summed over the text queries should be normalized to 1 and vice versa, so that in overall, all text and video instances are equally represented during retrieval. however, we show that in practice, cross-modal contrastive learning suffers from significant normalization errors of the sum retrieval probabilities of each instance (fig. 2 ). this suggests that many test instances are either over-or under-represented during retrieval, leading to high false positive and false negative rates (fig. 3 ) and consequently harming the text-video retrieval performance. to address this problem, we propose normalized contrastive learning (ncl) which computes instance-wise biases using the sinkhorn-knopp algorithm @xcite and adjusts the cross-modal embedding similarity scores so that the sum retrieval probabilities of each instance are properly normalized to 1 (fig. 1 ). at test time where the test queries are not known a priori, we show that we can approximate the test query distribution by storing a subset of train queries during training in a queue. we show that this approach consistently reduces the normalization errors (fig. 2 ) and thereby improves the text-video retrieval performance significantly. we evaluate ncl on text-video retrieval on the activitynet, msvd, and msr-vtt datasets. empirical results show that ncl consistently improves both text-to-video and video-to-text retrieval across all datasets on different base architectures, including those of clip4clip @xcite and ssb the instance-wise biases that normalize the sum retrieval probabilities so that all instances are fairly represented @xcite and further advances stateof-the-art text-video retrieval performance without any architecture engineering. in summary, our contributions are: ## related work contrastive representation learning infonce (noise contrastive estimation of mutual information) has been developed for learning unsupervised representations of natural images @xcite @xcite . it has quickly gained popularity due to its simplicity and effectiveness, leading the state-ofthe-art advances in visual representation learning. specifically, it first samples two different views of an image using random data augmentations. the views of a same image constitute a positive pair, while negative pairs consist of views of different images in a mini-batch. the encoder is then trained to minimize the cross-entropy loss to classify the positive pairs from the set of negative pairs based on the embedding similarity between the representations. recently, the scope of contrastive learning has been extended to multimodal data @xcite @xcite @xcite . in crossmodal contrastive learning, different modalities of the data are mapped to a shared embedding space using separate encoders for each modality, where their cross-modal similarity is defined. in particular, clip @xcite trains its image and text encoders on millions of image-text pairs using cross-modal contrastive learning and demonstrates that the learned representations can perform zero-shot transfer on various vision tasks. @xcite further improves the vision-language contrastive learning by incorporating additional language modeling and image-text matching losses. aside from infonce where each example is assigned its own class label, a recent line of works @xcite assume that the data can be clustered into k latent classes. they generate pseudo labels for the data by solving an optimal transport problem with entropy regularization, using the sinkhorn-knopp algorithm @xcite . the encoder is then trained to predict the pseudo-labels of the data. in contrast to previous work that used the sinkhorn-knopp algorithm for self-labeling, in this work we use the algorithm to compute the instance-wise biases that properly normalizes the sum retrieval probabilities of each text or video instance so that all instances are fairly represented during retrieval. embedding-based text-video retrieval maps video and text data into a shared multimodal embedding space where the similarity between a textvideo pair is defined as the cosine similarity of their embeddings. many recent works build upon pretrained vision and text encoders @xcite @xcite @xcite @xcite and finetune their models on downstream text-video datasets. for example, @xcite aggregates multiple expert features including objects, motion, appearance, and audio using a collaborative gating mechanism and minimizes ranking loss to align video and text embeddings for retrieval. similarly, @xcite applies self-attention to video expert features to get video-level representations. on the other hand, @xcite trains multiple teacher models using different pretrained text encoders and distills their knowledge to a student model. @xcite introduces an auxiliary task of reconstructing the caption of a video from other similar videos in a mini-batch to improve the learning of multimodal representation. @xcite tailors vision transformer architecture to train the model on both image and video together and applies curriculum learning by gradually increasing the number of frames the vision encoder takes. @xcite builds hierarchical transformers and performs hierarchical cross-modal contrastive matching at both the feature and semantic levels. in particular, clip4clip @xcite achieves state-of-the-art performance on text-video retrieval by loading the pretrained clip model @xcite and finetuning it on text-video datasets using cross-modal contrastive learning. however, we show that clip4clip significantly suffers from incorrect normalization of the sum retrieval probabilities of each instance and propose normalized contrastive learning to address this problem. ## experiments we evaluate normalized contrastive learning (ncl) on multimodal retrieval on popular textvideo datasets including activitynet, msvd, and msr-vtt and report recall-at-k (r@k) metrics (higher is better), median and mean rank (lower is better). the main goals of the experiments are: for a fair comparison, we assume that the test query distribution is not known in advance and use train query queues for test time normalization. datasets activitynet @xcite is a collection of 20,000 youtube videos. following @xcite , we concatenate the text descriptions of a video into one paragraph and perform video-paragraph retrieval on activitynet. we use the 'val1' split for evaluation which contains 5k videos and paragraphs. the train split has 10k video-paragraphs. msvd @xcite has 1,970 videos with each video having approximately 40 captions. the train split has 1200 videos, with 100 videos in the validation and 760 videos in the test split. in msvd, each video in the test set has multiple captions associated with it. msr-vtt @xcite consists of 10,000 videos with 20 text captions per each video. we use the 9k train split with 180k captions and the 1k test split following @xcite . the test split only contains one caption per video. architecture we use the state-of-the-art architecture of clip4clip @xcite with the code released by the authors. clip4clip is based on the pretrained clip model @xcite which consists of transformer-based vision and text encoders trained on large-scale image-text data. for the experiments, we adopt the mean pooling (meanp) architecture for aggregating the frame features as it was shown to deliver the most consistent performance on both text-to-video and video-totext retrieval across different datasets @xcite . we do not add any trainable model components and use the clip4clip architecture as-is. implementations details are described in appendix b. ## conclusion we've presented normalized contrastive learning (ncl) to improve cross-modal contrastive learning for retrieval. ncl applies the sinkhorn-knopp algorithm to normalize the retrieval probabilities of text and video instances so that each instance is fairly represented during retrieval. when the test queries are not known a priori, ncl approximates the test query distribution with a subset of train queries stored during training. empirical studies show that ncl consistently reduces the normalization errors and brings significant gains in stateof-the-art text-video retrieval performance without any architecture engineering. moreover, the gains are consistent over different model architectures. for future work, it will be worthwhile to explore if ncl can help retrieval tasks in other domains such as image and text, and whether it can improve general representation learning for downstream tasks such as classification.