MAF : Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding (2020)

Abstract: Phrase localization is a task that studies the mapping from textual phrases to regions of an image. Given difficulties in annotating phrase-to-object datasets at scale, we develop a Multimodal Alignment Framework (MAF) to leverage more widely-available caption-image datasets, which can then be used as a form of weak supervision. We first present algorithms to model phrase-object relevance by leveraging fine-grained visual representations and visually-aware language representations. By adopting a contrastive objective, our method uses information in caption-image pairs to boost the performance in weakly-supervised scenarios. Experiments conducted on the widely-adopted Flickr30k dataset show a significant improvement over existing weakly-supervised methods. With the help of the visually-aware language representations, we can also improve the previous best unsupervised result by 5.56%. We conduct ablation studies to show that both our novel model and our weakly-supervised strategies significantly contribute to our strong results.

## introduction language grounding involves mapping language to real objects or data. among language grounding tasks, phrase localization-which maps phrases to regions of an image-is a fundamental building block for other tasks. in the phrase localization task, each data point consists of one image and its corresponding caption, i.e., @xmath0, s , where i denotes an image and s denotes a caption. typically, the caption s contains several query phrases @xmath1, where each phrase is grounded to a particular object in the image. the goal is to find the correct relationship between (query) phrases in 1 code is available at https://github.com/ qinzzz/multimodal-alignment-framework . the caption and particular objects in the image. existing work @xcite @xcite @xcite ) mainly focuses on the supervised phrase localization setting. this requires a large-scale annotated dataset of phrase-object pairs for model training. however, given difficulties associated with manual annotation of objects, the size of grounding datasets is often limited. for example, the widely-adopted flickr30k @xcite dataset has 31k images, while the caption dataset ms coco @xcite ) contains 330k images. to address this limited data challenge, two different approaches have been proposed. first, a weakly-supervised setting-which requires only caption-image annotations, i.e., no phrase-object annotations-was proposed by @xcite . this is illustrated in figure 1 . second, an unsupervised setting-which does not need any training data, i.e., neither caption-image and phraseobject annotation-was proposed by @xcite . to bring more semantic information in such a setting, previous work @xcite used the detected object labels from an off-the-shelf object detector (which we will generically denote by predet) and achieved promising results. in more detail, for a given im-an older gentleman is standing next to the man with a red accordion over his shoulder. age i, the predet first generates a set of objects @xmath2. afterward, all the query phrases p and the detected objects o are fed into an alignment model to predict the final phrase-object pairs. however, purely relying on the object labels causes ambiguity. for example, in figure 2 , the grounded objects of phrases "an older man" and "the man with a red accordion" are both labeled as "man," and thus they are hard to differentiate. given these observations, we propose a multimodal alignment framework (maf), which is illustrated in figure 3 . instead of using only the label features from the predet (in our case, a faster r-cnn @xcite )), we also enhance the visual representations by integrating visual features from the faster r-cnn into object labels. (this is shown in figure 2 .) next, we build visually-aware language representations for phrases, which thus could be better aligned with the visual representations. based on these representations, we develop a multimodal similarity function to measure the caption-image relevance with phrase-object matching scores. furthermore, we use a training objective to score relevant captionimage pairs higher than irrelevant caption-image pairs, which guides the alignment between visual and textual representations. we evaluate maf on the public phrase localization dataset, flickr30k entities @xcite . under the weakly-supervised setting (i.e., using only caption-image annotations without the more detailed phrase-object annotations), our method achieves an accuracy of 61.43%, out-performing the previous weakly-supervised results by 22.72%. in addition, in the unsupervised setting, our visually-aware phrase representation improves the performance from the previous 50.49% by 5.56% up to 56.05%. finally, we validate the effectiveness of model components, learning methods, and training techniques by showing their contributions to our final results. ## related work with the recent advancement in research in computer vision and computational linguistics, multimodal learning, which aims to explore the explicit relationship across vision and language, has drawn significant attention. multimodal learning involves diverse tasks such as captioning @xcite @xcite , visual question answering @xcite @xcite , and vision-and-language navigation @xcite @xcite . most of these tasks would benefit from better phrase-to-object localization, a task which attempts to learn a mapping between phrases in the caption and objects in the image by measuring their similarity. existing works consider the phrase-to-object localization problem under various training scenarios, including supervised learning @xcite @xcite @xcite and weakly-supervised learning @xcite @xcite . besides the standard phrase-object matching setup, previous works @xcite @xcite have also explored a pixellevel "pointing-game" setting, which is easier to model and evaluate but less realistic. unsupervised learning was studied by @xcite , who directly use word similarities between object labels and query phrases to tackle phrase localization without paired examples. similar to the phrase-localization task, @xcite leverages document-level supervision to discover image-sentence relationships over the web. final output feature of predet (denoted as f m ) as the vfr, and wang and specia (2019) uses the label embedding (denoted as l m ) of the predicted label from predet as the vfr. this unitary vfr usually lacks the counter-side information. hence, we exploit different aspects of features extracted from predet for each object o m in the image. in particular, we consider the output feature f m , the label embedding l m , and the attribute embedding t m of the object o m as the vfr, ## empirical results dataset details. the flickr30k entities dataset contains 224k phrases and 31k images in total, where each image will be associated with 5 captions and multiple localized bounding boxes. we use 30k images from the training set for training and 1k images for validation. the test set consists of 1k images with 14,481 phrases. our evaluation metric is the same as plummer et al. (2015). 2 we consider a prediction to be correct if the iou (intersection of union) score between our predicted bounding box and the ground-truth box is larger than 0.5. following @xcite , if there are multiple ground-truth boxes, we use their union regions as a single ground-truth bounding box for evaluation. weakly-supervised results. we report our weakly-supervised results on the test split in resnet-101 61.43 86.29 unsupervised results. 3 we report our unsupervised results for the phrase localization method (described in section 3.2) in method tfr detector acc. (ub) (%) whole image none none 21.99 (wang and specia, 2019) w2v-max faster r-cnn 49.72 (86.29) (wang and specia, 2019) w2v-max cc+oi+pl 50.49 (57.81) maf (ours) glove-att faster r-cnn 56.05 (86.29) ablation experiments. in this section, we study the effectiveness of each component and learning strategy in maf. the comparison of different feature representations is shown in we then investigate the effects of different initialization methods for the two weight matrices, w f and w p . the results are presented in ## conclusions we present a multimodal alignment framework, a novel method with fine-grained visual and textual representations for phrase localization, and we train it under a weakly-supervised setting, using a contrastive objective to guide the alignment between visual and textual representations. we evaluate our model on flickr30k entities and achieve substantial improvements over the previous state-of-the-art methods with both weakly-supervised and unsupervised training strategies. detailed analysis is also provided to help future works investigate other critical feature enrichment and alignment methods for this task.