Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding (2021)

Abstract: Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying objects based on language. With pre-trained vision-and-language models achieving impressive performance across tasks, it remains unclear if we can directly utilize their learned embeddings for phrase grounding without fine-tuning. To this end, we propose a method to extract matched phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to improve the model phrase grounding ability using image-caption data without any supervised grounding signals. Experiments on two representative datasets demonstrate the effectiveness of our objectives, outperforming baseline models in both weakly-supervised and supervised phrase grounding settings. In addition, we evaluate the aligned embeddings on several other downstream tasks and show that we can achieve better phrase grounding without sacrificing representation generality.

## introduction recent studies on vision-and-language pretraining @xcite @xcite @xcite @xcite demonstrate impressive performance across vision-andlanguage tasks, including image-text retrieval @xcite , visual entailment @xcite and visual question answering @xcite . however, few existing papers have paid attention to the phrase grounding ability of their pretrained embeddings, namely the ability to map natural language queries to their corresponding image regions, which can 1) benefit tasks requiring identifying objects based on language @xcite ; 2) be a prerequisite for advanced multimodal reasoning @xcite . among the prior work, @xcite demonstrate certain grounding abilities of visualbert, yet their analysis is limited to attention heads and it is unclear how vi-sualbert compares with state-of-the-art grounding models. @xcite provide insights on cross-modal interaction, but their analysis is primarily limited to the coreference relations between phrases and visual tokens. in this paper, we study the phrase grounding ability of vision-and-language embeddings pre-trained on image-caption datasets. first, we propose a method to extract phrase-region pairs from the pre-trained embeddings without any fine-tuning. we find that while our method uncovers certain grounding abilities of the pre-trained embeddings, there is still much room for improvement. therefore, we propose to fine-tune models with objectives designed for better aligning word and region representations on image-caption datasets. the fine-tuning objectives are designed to maximize the symmetricity between vision and language during fine-tuning for better phrase grounding while maintaining the representation transferability so that the learned representations are still useful for other downstream tasks. specifically, we fine-tune models with 1) a masked language modeling objective conditioned on images; 2) an adapted masked region modeling objective with texts utilizing a dynamically constructed vision vocabulary; 3) a modified object label prediction objective that explicitly bridges the gap between vision and language; 4) a proposed bidirectional attention optimization objective encouraging the consistency between visionto-language and language-to-vision alignments. we fine-tune pre-trained models on coco @xcite and test them on two representative phrase grounding datasets, refcoco+ @xcite and flickr30k entities @xcite . we find that our fine-tuning objectives can improve the model grounding ability significantly, improving baseline in both weakly-supervised and supervised phrase grounding settings. we also evaluate the aligned representations on several downstream tasks and show that our model can achieve better phrase grounding without sacrificing performance on other types of tasks. ## extracting phrase-region pairs from pre-trained embeddings formally, the phrase grounding task can be defined as: given an image v consisting of multiple re- we first propose a way to directly extract the matched phrase-region pairs from pre-trained embeddings. then, we evaluate this method on phrase grounding tasks with several popular pre-trained models, including lxmert @xcite , uniter @xcite , vilbert @xcite , visualbert @xcite and vl-bert @xcite . ## aligning pre-trained vision-and-language embeddings to improve the model phrase grounding ability, we then propose four fine-tuning objectives for vision-and-language models. we assume an imagecaption dataset { v k , l k } is provided but no finegrained phrase-region annotations are available. a pre-trained vision model @xcite is used to segment images into regions and produce region representations and object labels. ## related work we overview two lines of related work in this part. ## conclusion in this paper, we first propose a method to extract matched phrase-region pairs from pre-trained vision-and-language embeddings and evaluate its performance across models. then, we propose several fine-tuning objectives for phrase grounding and demonstrate their effectiveness in both weakly-superivsed and supervised phrase grounding tasks. we also evaluate our aligned representations on other downstream tasks and show that we can achieve better phrase grounding without sacrificing the representation transferability to other downstream tasks. future directions include better utilizing the aligned representations and incorporating our objectives into pre-training. qinxin wang, hao tan, sheng shen, michael mahoney, and zhewei yao. 2020. maf: multimodal alignment framework for weakly-supervised phrase grounding. in proceedings of the conference on empirical methods in natural language processing (emnlp). ning xie, farley lai, derek doran, and asim kadav. 2019. visual entailment: a novel task for finegrained image understanding. arxiv preprint. yujia xie, xiangfeng wang, ruijia wang, and hongyuan zha. 2020. a fast proximal point method for computing exact wasserstein distance. in proceedings of the conference on uncertainty in artificial intelligence (uai). raymond a yeh, minh n do, and alexander g schwing. 2018. unsupervised textual grounding: linking words to image concepts. in proceedings of the ieee conference on computer vision and pattern recognition (cvpr). zhou yu, jun yu, chenchao xiang, zhou zhao, qi tian, and dacheng tao. 2018. rethinking diversified and discriminative proposal generation for visual grounding. in proceedings of the international joint conferences on artificial intelligence organization (ijcai).