Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023)

Abstract: In this paper, we reconsider the problem of (partial) false negative samples from the Mutual Information (MI) Maximization perspective, the traditional contrastive loss (like InfoNCE loss) will equally push away the anchor of all positive samples and negative samples regardless of their possible semantic similarities. We theoretically show that InfoNCE loss will not only maximize the MI between the anchor and positive samples but minimize the MI between the anchor and false negative samples even though they share similar semantic which could provide a possible theoretical explanation for the observation of the existence of false negative samples in the cross-modal contrastive learning will decrease the downstream task performance of VLP models. Above analysis motivate us to propose the VLP model with a novel Semantic Awared Contrastive Learning framework named SACL where different negative samples are assigned with different contrastive weights according to the semantic similarity between them and the anchor.

## introduction large-scale pre-trained vision-language models have recently achieved tremendous success on a wide range of cross-modal tasks @xcite @xcite @xcite @xcite @xcite . self-supervised learning (ssl) @xcite have impressively contributed to vision-language pretraining (vlp) due to its capability of leveraging large-scale image-text pairs without annotations. more recently, self-supervised multi-modal contrastive learning (smcl) triggered great progress @xcite @xcite @xcite by conducting crossmodal alignment. smcl consists of image-totext and text-to-image contrastive learning, e.g., * corresponding author. , negative samples will be equally pushed away from the anchor in conventional cross-modal contrastive learning, leading to data deficiency given these false ones. instead, we take the first step to contrast negatives according to cross-modal similarity (represented by a set of concentric circles in sub-figure (c)), keeping good while removing harmful effects of (partial) false negative samples. with the infonce @xcite loss. taking the text-to-image one as an example, given a text-image pair (t, i), i will be treated as the positive sample for the anchor t, and other images in a mini-batch of text-image pairs will be regarded as negatives. the training objective is to attract the positive to the anchor while repelling all the negative samples. however, this contrasting strategy can be problematic given the many-to-many correspondences between images and texts. as shown in figure 1 (a), a text can be semantically paired with multiple images. in this scenario, though images i 4 and i 5 are treated as negatives, they are actually semantically consistent (or partially consistent) with the text anchor "a bird in the tree." the (partial) false negatives like i 4 and i 5 will inevitably hinder the contrasting effect, yielding sub-optimal crossmodal representations. some pioneering efforts have addressed the noisy image-text pairing problem in vlp pretraining datasets @xcite , by feeding the contrastive loss with soft labels in a self-distillation manner. though these methods can address the problem of false negatives to some extent, the specific harmful effect of false negatives remains far from being systematically studied. for example, based on these methods (e.g., albef @xcite ), we can easily improve the performances of downstream tasks by simply filtering false negatives, as shown in in this paper, we investigate the problem of false negatives from the perspective of mutual information (mi) optimization. the infonce loss used in contrastive learning has been proved to maximize the lower bound of mi between anchors and their positives @xcite . we revisit the theoretical proof in the presence of non-negligible false negatives. defining the mi between anchors and positives as mi-p, and the counterpart between anchors and negatives as mi-n, we derive a more general conclusion (see the appendix a.2) that optimizing infonce is equivalent to maximizing the lower bound of (mi-p -mi-n). the finding suggests that mi-n will be minimized (e.g., as close to zero as possible), even though some negatives may semantically match the anchor. the theoretical analyses explain the deficiency of the vanilla contrasting strategy on the one hand, and inspire us with another derivation (appendix a.3) that guarantees proper mi optimization for negative samples on the other hand. guided by these theoretical analyses, we propose a novel contrasting strategy regulated by crossmodal similarity. we hypothesize that the mi between an image and text positively correlates with their semantic similarity. therefore, we introduce a contrastive weight, which is derived based on cross-modal similarity and progressively refined with training, for each negative sample as a contrasting regulator. this regulator will guide the model to optimize mi-n properly, keeping it from being unexpectedly minimized and thus yielding a more semantically structural representation space. we equip our proposed contrasting strategy on al-bef @xcite , we directly remove false negatives samples in a heuristic way from a mini-batch (more details in section 4.3), achieving a new pre-trained model albef++. we report the performance of zero-shot cross-modal retrieval (flicker 30k) and visual question answering (vqa). even by simply removing false negatives, albef++ outperforms albef by an evident margin, indicating that existing efforts have not sufficiently addressed the harmful effects of false negatives. cross-modal retrieval, zero-shot cross-modal retrieval, and natural language for visual reasoning (nlvr). the experimental results show that our adjusted contrastive learning significantly improves their performances. in summary, our contributions are: ## theoretical analysis from mutual information perspective mutual information (mi) is designed to measure the relationship between random variables or determine the amount of shared information @xcite . oord et al. (2018) has proven that the infonce loss function widely used in contrastive learning can be seen as a lower bound of mi between anchors and positives. note that li et al. (2021) provides a conceptual yet more intuitive discussion of the correspondence between infonce and mi in the vlp scenario. in this paper, we go one step further to revisit the proof of oord et al. (2018) under a cross-modal contrastive learning context. ## method in this section, we will first introduce our model architecture, and then introduce our similarityseparately and a multi-modal encoder for the fusion of the cross-modal information. after feeding the input to the unimodal encoders, we take the representation of [cls] token as the global representation and use similarity-regulated contrastive learning (srcl) to align the unimodal representations of an image-text pair. we also apply an image-text matching loss and a masked-language-modeling loss to learn multimodal interactions between image and text. regulated contrastive learning (srcl), followed by the details of other pre-training objectives. ## conclusion we have presented our cross-modal contrastive learning method that addresses the problem of (partial) false negatives with vision-language semantic similarity guidance. a series of mathematical proofs based on infonce loss provides a more general lower bound for contrastive optimization and inspires us with a novel contrasting strategy that theoretically guarantees the mitigation of false negatives. empirically, our method demonstrates performance superiority on four downstream crossmodal tasks. meanwhile, by comparing false negatives and hard negatives, we reveal that balancing the beneficial and harmful effects of (partial) false negatives is crucial to learn robust cross-modal representations.