Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding (2022)

Abstract: Weakly supervised phrase grounding aims to learn an alignment between phrases in a caption and objects in a corresponding image using only caption-image annotations, i.e., without phrase-object annotations. Previous methods typically use a caption-image contrastive loss to indirectly supervise the alignment between phrases and objects, which hinders the maximum use of the intrinsic structure of the multimodal data and leads to unsatisfactory performance. In this work, we directly use the phrase-object contrastive loss in the condition that no positive annotation is available in the first place. Specifically, we propose a novel contrastive learning framework based on the expectation-maximization algorithm that adaptively refines the target prediction. Experiments on two widely used benchmarks, Flickr30K Entities and RefCOCO+, demonstrate the effectiveness of our framework. We obtain 63.05% top-1 accuracy on Flickr30K Entities and 59.51%/43.46% on RefCOCO+ TestA/TestB, outperforming the previous methods by a large margin, even surpassing a previous SoTA that uses a pre-trained vision-language model. Furthermore, we deliver a theoretical analysis of the effectiveness of our method from the perspective of the maximum likelihood estimate with latent variables.

## introduction phrase grounding aims to localize corresponding objects in an image given a phrase in the image's caption. it is one of the most fundamental research areas in multimodal learning @xcite . this area has strong applications in other complex visual language tasks, such as visual question answering @xcite , crossmodal retrieval @xcite , etc. current methods @xcite have achieved great success but rely heavily on bounding box annotations, which are expensive to acquire. thus, weakly supervised phrase grounding has recently received increased attention @xcite @xcite due to the low cost of obtaining image-text pair annotations. it aims to ground phrases using only caption-image annotations, i.e., without phrase-object annotations. ## related work weakly supervised phrase grounding weakly supervised phrase grounding has received considerable attention, as the fully supervised setting requires the labelling of phrases in image captions. previous mainstream methods can be divided into two main categories: reconstruction-based methods @xcite @xcite and contrastive-based methods @xcite @xcite @xcite . our work falls into the line of contrastive-based methods. previous contrastive-based methods typically conduct contrastive learning at the caption-image level by heuristically aggregating several phraseobject similarities to be caption-image similarities. align2ground @xcite aggregates several features of objects to get a caption-conditioned image representation, and matches it with the corresponding caption. infoground @xcite defines a compatibility function to measure the compatibility between images and bertcontextualized word representation. infoground uses bert to generate hard negative examples. ccl @xcite defines an aggregation function to compute the alignment score between a phrase and a set of objects, and generates counterfactual examples by the gradient. maf @xcite uses the mean of phrase-wise maximum similarity as the caption-image similarity and performs contrastive learning at the caption-image level. these works are distinctively different in their definition of the aggregation function and the approach in which they generate positive and negative examples. none of these works use contrastive learning at the phrase-object level nor do they highlight the importance of intrinsic structure as we do. contrastive learning contrastive learning has recently attracted much attention, as it has contributed to the success in unsupervised representation learning. there exist a line of work that explore contrastive learning at the instance level in computer vision, natual language processing and multimodal learning. for example, moco @xcite , simclr @xcite , simcse @xcite , clip @xcite , and albef @xcite . another line explore the learning problem in a weakly supervised setting. pico @xcite is one of such works, which uses prototypes to address label disambiguation in partial label learning. our work is closely related to pico. however, our work differs from pico in twofold: firstly, our model is oriented toward a multimodal setting while pico only works with a unimodal setting (i.e., image). secondly, pico only suits classification problems where the total number of labels is fixed while our model is unconstrained to the number of labels. specifically, in phrase grounding, the total number of possible objects is unlimited. hence, pico cannot be directly applied to this task. ## problem formulation given a caption-image pair (s, i), a grounding model is expected to find the object k j among m objects in image i which refers to the given phrase q i in caption s: it is difficult to solve this objective due to the lack of phrase-region annotations. therefore, we treat this annotation as a latent variable z i . that is, the phrase q i refers to the region k j when z @xmath0. we then solve the following maximum likelihood estimate problem instead: note that @xcite solve log p(k j |s). on the contrary, we solve max log p(q i |i). this is based on the observation that all phrases are conditioned on images but not all regions have a corresponding phrase. the maximum likelihood estimate of p(k j |s) decreases the alignment performance when k j has no corresponding phrase in sentence s. a model that solves (2) provides a competitive estimate for latent variable z i . for clarity, we give several definitions here. with respect to a given query phase, an image region falls into one of the three categories: true positive, false positive, and (true) negative. all regions in an image not paired with the phase are regarded as negatives. all regions in the image paired with the phrases are positives, in which only the one that semantically corresponds to the phrase is the true positive and the rest are false positives. ## methodology in this section, we introduce the proposed contrastive learning framework. we follow a 2-stage paradigm. in the first stage, we apply a pre-trained object detector to extract object features. in the second stage, we sort the objects by their similarities with phrases. as shown in figure 2 , our framework consists of three main components: image encoder f k : it extracts bounding boxes and features of objects in an image. we let k 1 , k 2 , . . . , k @xmath1, where i is an image and m is the number of objects in the image. text encoder f q : it takes as input a caption s and outputs n phrase features, denoted by q 1 , q 2 , . . . , q @xmath2. contrastive loss module: it adaptively refines the alignment score between objects and phrases. the key idea is to apply an e-step and m-step iteratively for this purpose. in the following subsections, we describe each component in detail. ## experiments here, we present our experiments and results. ## conclusion in this paper, we identify the significance of the intrinsic similarity and propose a novel contrastive learning framework based on the expectationmaximization algorithm to solve weakly supervised phrase grounding. our method can adaptively refine the model prediction by minimizing a contrastive loss and chronologically updating pseudo targets. to validate our method, we conduct extensive experiments and deliver a theoretical analysis from the perspective of the maximum likelihood estimate. our proposed approach achieves state-of-the-art performance on two widely used datasets, filckr30k entities and refcoco+. for future works, a more sophisticated language model is worth to be explored for phrase modeling to associate complex phrases with an object. we are also interested in investigating to what extent our