Extending Phrase Grounding with Pronouns in Visual Dialogues (2022)

Abstract: Conventional phrase grounding aims to localize noun phrases mentioned in a given caption to their corresponding image regions, which has achieved great success recently. Apparently, sole noun phrase grounding is not enough for cross-modal visual language understanding. Here we extend the task by considering pronouns as well. First, we construct a dataset of phrase grounding with both noun phrases and pronouns to image regions. Based on the dataset, we test the performance of phrase grounding by using a state-of-the-art literature model of this line. Then, we enhance the baseline grounding model with coreference information which should help our task potentially, modeling the coreference structures with graph convolutional networks. Experiments on our dataset, interestingly, show that pronouns are easier to ground than noun phrases, where the possible reason might be that these pronouns are much less ambiguous. Additionally, our final model with coreference information can significantly boost the grounding performance of both noun phrases and pronouns.

## introduction grounded language learning has been prevailing for decades in many fields @xcite , generally aiming to learn the real-world meaning of textual units (e.g., words or phrases) by conjointly leveraging the perception data (e.g., images or videos). @xcite advocate that we cannot overlook the physical world that language describes when doing language understanding research from a novel perspective. in particular, with the stimulation of modeling techniques and multimodal data collection paradigms, the task has made excellent progress in the downstream tasks, which involves multi-modal question answering @xcite , video-text align-figure 1 : an example of grounding noun phrases and pronouns referred in the caption and dialogue (partly) to the associated image regions. with an image described by a caption, two people are discussing what they can see. here, we annotate the same object with the same color, respectively. and obviously, the same object mentioned in the text naturally forms a coreference chain. ment @xcite and robot navigation @xcite . typically, as one branch of grounded language learning, phrase grounding, first proposed by @xcite , also plays a key role in visual language understanding. its goal is to ground the phrases in a given caption to the corresponding image regions. recently, many researchers have attempted varied approaches to explore this task. @xcite propose a novel graph learning framework for phrase grounding to distinguish the diversity of context among phrases and image regions. @xcite develop a multimodal alignment framework to utilize the caption-image datasets under weak supervision. @xcite advance phrase grounding with their end-to-end modulated pre-trained network named mdetr. overall, the natural language processing (nlp) and computer vision (cv) communities have seen huge achievements in the task of phrase grounding. in spite of its apparent success, there remains a worth-thinking weakness. almost all previous works mainly focus on the noun phrases/words, which can derive their meanings by the expressional forms to some extent. there is little work that takes account into pronouns. as shown in fig -ure 1, pronouns definitely have underlying effects on the performance of visual grounding, which should be carefully examined @xcite . as a result, here we shift our eyes from the common (almost noun) phrase grounding with the extension of pronouns for the first time. in this paper, we present the first work for investigating phrase grounding that includes pronouns, and explore how coreference chains can have an effect on the performance of our task. we annotate an initial dataset based on visual dialogue @xcite , as shown in figure 1 . for the model, we can directly apply mdetr @xcite , which is an end-to-end modulated detector. however, the model does offer much information to understand pronouns. thus, we enhance the vanilla model with coreference information from the dialogue end, where a graph neural network is adopted to encode the graph-style coreference knowledge. finally, we conduct experiments on our constructed dataset to benchmark the extended phrase grounding task. according to the results, we find that interestingly, pronouns are easier to ground by mdetr than phrases. the underlying reason might be that the pronouns are always more important during dialogue, leading to less ambiguity in speech communication. in addition, our final model can be significantly enhanced by adding the gold graph-style coreference knowledge; however, the model fails to obtain any positive gain when the coreference information is sourced from a stateof-the-art machine learning model. we conduct several in-depth analyses for comprehensive understanding of our task as well as the model. in summary, our contributions are as follows: ## method recent works @xcite bring the successful vision-language transformer architecture and the pre-train-then-finetune paradigm to the phrase grounding task, achieving state-ofthe-art performance. to explore our constructed dataset, we adopt the representative mdetr (kamath et al., 2021) model. meanwhile, we propose to enhance the textual representations with the natural coreference chains in texts by relational graph convolutional networks (r-gcn) @xcite . bellow, we briefly describe how mdetr learns and grounds, and then present our suggested coreference graph encoding. ## related work visual grounding. general visual grounding, also known as referring expression comprehension @xcite , is akin to phrase grounding to some extent, since they all aim to study the mapping from the expressions to the specific image regions. the main difference between them is that the visual grounding particularly focuses on one single expression, while the phrase grounding is more general and can be applied to multiple expressions. ## conclusion in this work, we proposed to extend phrase grounding task with pronouns, additionally, we established our dataset, vd-ref, the first dataset which contains ground-truth mappings from both noun phrases and pronouns to image regions. furthermore, we took the state-of-the-art model mdetr as our baseline and introduced extra coreference knowledge with graph neural networks. experiments on our dataset showed the exciting phenomenon that pronouns are more accessible grounded than phrases and demonstrated the significance of coreference knowledge in visual language understanding. to this end, we conducted in-depth analyses of our results. in the future, we would expand more sophisticated dataset, and do more richer experiments on our dataset. our dataset and baseline code are avaliable at https://github.com/izhx/phrase-grounding-withpronoun .