Flexible Visual Grounding (2022)

Abstract: Existing visual grounding datasets are artificially made, where every query regarding an entity must be able to be grounded to a corresponding image region, i.e., answerable. However, in real-world multimedia data such as news articles and social media, many entities in the text cannot be grounded to the image, i.e., unanswerable, due to the fact that the text is unnecessarily directly describing the accompanying image. A robust visual grounding model should be able to flexibly deal with both answerable and unanswerable visual grounding. To study this flexible visual grounding problem, we construct a pseudo dataset and a social media dataset including both answerable and unanswerable queries. In order to handle unanswerable visual grounding, we propose a novel method by adding a pseudo image region corresponding to a query that cannot be grounded. The model is then trained to ground to ground-truth regions for answerable queries and pseudo regions for unanswerable queries. In our experiments, we show that our model can flexibly process both answerable and unanswerable queries with high accuracy on our datasets.

## introduction starting from conventional vision-and-language tasks such as image captioning @xcite and visual question answering @xcite , many studies have been conducted to promote joint vision-and-language understanding. visual grounding, which aims to find a specific region in an image given a query regarding an entity, is a fundamental task for enhancing the performance of various joint vision-and-language tasks @xcite . for instance, in image captioning, it is important to ground to the corresponding image region while generating words for that region; in in previous work, a query must be able to be grounded (see the left sub-figure), while our work can deal with both answerable and unanswerable visual grounding flexibly (in the right sub-figure, "two wonderful horses" can be grounded, while "my favorite picture," "a beautiful sunrise," and "a frosty day" cannot be grounded). the green bounding boxes are the ground-truth for answerable queries. vqa, it is crucial to understand to which image region the question is referring. because of the importance of visual grounding, many research efforts have been dedicated to improve its accuracy @xcite @xcite @xcite @xcite @xcite @xcite . previous visual grounding work assume that a query must be able to be grounded to an image region and create many datasets such as the flickr30k entities @xcite , refclef @xcite , refcoco, refcoco+ @xcite , refcocog @xcite , and visual7w datasets @xcite for the task. however, this assumption is not true in realworld multimedia data such as news, tv dramas, and social media, where entities in the text are not always able to be grounded to the visual data due to the fact that text and visual data in these multimedia data are unnecessarily directly corresponding to each other. we name the case that a query can be grounded to an image region as answerable visual grounding; otherwise, unanswerable visual grounding from here. the ignorance of unanswerable visual grounding in previous work can lead to problems for downstream tasks. for instance, in vqa, if the vqa model cannot understand the case that entities in the question cannot be grounded to the image, it cannot deal with the case that a question cannot be answered given the image either. therefore, a robust visual grounding model should be able to flexibly deal with both answerable and unanswerable visual grounding. in this work, we study this flexible visual grounding problem. figure 1 compares our work with previous work. to study flexible visual grounding, we construct two types of datasets. the first one is a pseudo dataset, which is constructed by randomly selecting queries from other images and combining it with a target image in the refcoco+ dataset @xcite . the second one is a social media dataset (smd4fvg), which contains unanswerable realworld queries. we construct the smd4fvg dataset by crawling tweets consisting of both images and text and annotating answerable and unanswerable queries via crowdsourcing. previous visual grounding models cannot handle unanswerable visual grounding. to give a model the ability to flexibly identify whether the input query can be grounded or not, we propose a novel method for unanswerable visual grounding by adding a pseudo region corresponding to a query that cannot be grounded. the model is then trained to ground to ground-truth regions for answerable queries and pseudo regions for unanswerable queries. experiments conducted on both the pseudo and smd4fvg datasets indicate that our model can flexibly process both answerable and unanswerable queries with high accuracy. in addition, we study the possibility of the usage of using the pseudo dataset to improve the accuracy on the smd4fvg dataset. the contributions of this paper are in three-folds: ## related work previous visual grounding studies have been conducted on different datasets. in the flickr30k entities dataset @xcite , a query corresponds to a noun phrase (i.e., entity) containing in a caption of an image. in the refclef @xcite , refcoco, refcoco+ @xcite , and refcocog @xcite datasets, a query is an phrase referring to an object in an image. in the visual7w dataset @xcite , a query corresponds to a question regarding an image region. however, all these datasets do not consider unanswerable visual grounding. in contrast, we propose flexible visual grounding and construct a pseudo dataset and a social media dataset. regarding visual grounding models, @xcite proposed a method based on canonical correlation analysis @xcite ) that learns joint embeddings of phrases and image regions. @xcite proposed a two-branch neural network for joint phrasal and visual embeddings. @xcite used multimodal compact bilinear pooling to fuse phrasal and visual embeddings. @xcite proposed a method to first detect a candidate region for a given phrase and then reconstruct the phrase using the detected region. @xcite proposed an agreement-based method, which encourages semantic relations among phrases to agree with visual relations among regions. @xcite proposed a framework that can search over all possible regions instead of a fixed number of region proposals. @xcite used spatial relationships between pairs of phrases connected by verbs or prepositions. @xcite proposed a reinforcement learning-based model that rewards the grounding results with image-level context. @xcite improved the region proposal network by training it on the visual genome dataset @xcite to increase the diversity of object classes and attribute labels. @xcite proposed to combine object detection and grounding models to deal with unseen nouns during training. @xcite propagated relations among noun phrases in a query based on the linguistic structure of it. @xcite addressed the long and complex queries by recursive sub-query construction. @xcite proposed a crosslingual visual grounding task, which transfers the knowledge from an english model to improve the performance of a french model. inspired by the success of pre-training language models such as bert @xcite , visionand-language pre-training on large image caption datasets such as the conceptual captions dataset @xcite has been proposed such as vilbert @xcite vl-bert @xcite , and uniter @xcite . those vision-and-language pre-training models differ from the model architecture. visionand-language pre-training is evaluated on tasks including visual grounding. however, same to previous studies, the visual grounding task does not consider unanswerable cases @xcite @xcite . our flexible visual grounding model is based on the multi-task vilbert model @xcite , which achieves state-of-the-art performance on visual grounding. ## dataset construction because there are no existing visual grounding datasets where unanswerable queries are contained, we present two ways to construct two types of datasets to study the flexible visual grounding problem. ## flexible visual grounding model we propose to add a pseudo region to a visual grounding model to achieve flexible visual grounding for both answerable and unanswerable queries. an overview of our proposed model is shown in figure 3 . in this section, we first present our visual grounding model, followed by the way to add pseudo regions for unanswerable queries. ## experimental settings in our experiments, we verify the effectiveness of the proposed model on both the refcoco+ pseudo and smd4fvg datasets. here, we first describe the statistics of each dataset and settings, followed by training details. ## conclusion previous studies on visual grounding ignored the case of unanswerable queries, which is common in real-world such as social media data. in this paper, we proposed flexible visual grounding to address both answerable and unanswerable visual grounding. to this end, we constructed a pseudo dataset based on the refcoco+ dataset and a social media dataset based on tweets consisting of both images and text via crowdsourcing. in addition, we proposed a flexible visual grounding model, which can deal with both answerable and unanswerable queries. experiments on our datasets indicated that our model could achieve high accuracy, especially for unanswerable queries, but there is still room for further improvement. to make our social media dataset balanced, we constrained it to the refcoco+ classes, which may also limit the ability of our model on realworld data. in the future, we plan to construct a dataset without such constraints.