Modeling Coreference Relations in Visual Dialog (2021)

Abstract: Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. The occurrences of coreference relations in the dialog makes it a more challenging task than visual question-answering. Most previous works have focused on learning better multi-modal representations or on exploring different ways of fusing visual and language features, while the coreferences in the dialog are mainly ignored. In this paper, based on linguistic knowledge and discourse features of human dialog we propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way. Experimental results on the VisDial v1.0 dataset shows that our model, which integrates two novel and linguistically inspired soft constraints in a deep transformer neural architecture, obtains new state-of-the-art performance in terms of recall at 1 and other evaluation metrics compared to current existing models and this without pretraining on other vision language datasets. Our qualitative results also demonstrate the effectiveness of the method that we propose.

## introduction recently, with the unprecedented advances in computer vision and natural language processing, we have seen a considerable effort in developing artificial intelligence (ai) agents that can jointly understand visual and language information. visuallanguage tasks, such as image captioning @xcite and visual question-answering (vqa) @xcite , have achieved inspiring progress over the past few years. however, the applications of these agents in real-life are still quite limited, since 1 our code are released on: https://github.com/mingxiaoli/modeling-coreference-relations-in-visual-dialog they cannot handle the situation when continuous information exchange with a human is necessary, such as in visual-language navigation @xcite and visual dialog @xcite . the visual dialog task can be seen as a generalization of vqa. both tasks require the agent to answer a question expressed in natural language about a given image. a vqa agent needs to answer a single question, while a dialog agent has to answer a series of language questions based on its understanding of visual content and dialog history. compared to vqa, the visual dialog task is more difficult because it demands the agent to resolve visual coreferences in the dialog. considering the example in figure 1 . when the agent encounters question 6 "do they look old or new ?" and questions 9 "are they tall or short ?", it has to infer that the pronoun "they" in these two questions refers to different entities in the image or the dialog history. this paper studies how we can improve the results of a visual dialog task by better resolving the coreference relations in the dialog. in this work we restrict coreference resolution to pronouns. we use a multi-layer transformer encoder as our baseline model. based on the assumption that the output contextual embedding of a pronoun and its antecedent should be close in the semantic space, we propose several soft constraints that can improve the model's capability of resolving coreferences in the dialog in an unsupervised way (i.e., without ground truth coreference annotations in the training data). our first soft constraint is based on the linguistic knowledge that the antecedent of a pronoun can only be a noun or noun phrase. to integrate this constraint in the baseline model, we introduce a learnable part-of-speech (pos) tag embedding and a part-of-speech tag prediction loss. inspired by the observation that in human dialog the referents of pronouns often occur in nearby dialog utterances, we propose a second soft constraint using a sinusoidal sentence position embedding, which aims to enhance local interactions between nearby sentences. our contributions are as follows: first, as a baseline we adapt the multi-layer transformer encoder to the visual dialog task and obtain results comparable to the state of the art. second, we propose two soft constraints to improve the model's ability of resolving coreference relations in an unsupervised way. we also perform an ablation study to demonstrate the effectiveness of the introduced soft constraints. third, we conduct a qualitative analysis and show that the proposed model can resolve pronoun coreferents by making sure that in the neural architecture the pronoun mostly attends to its antecedent. ## related work visual dialog. the visual dialog task is proposed by @xcite , where a dialog agent has to answer questions grounded in an image based on its understanding of the dialog history and the image. most of previous work focuses on using an attention mechanism to learn interactions between image, dialog history and question. @xcite use an attention network to conduct multi-step reasoning in order to answer a question. @xcite propose a recursive attention network, which selects relevant information from the dialog history recursively. @xcite apply a multi-head attention mechanism @xcite to learn mutimodal representations. @xcite fuse information from all entities including question, answer, dialog history, caption and image using a factor graph. @xcite propose two-stage training. they first pretrain their transformer based two-stream attention network on other visual-language datasets, then finetune it on the visual dialog dataset. other approaches consider different learning methodologies to model the visual dialog task, for example, @xcite use adversarial learning @xcite apply reinforcement learning. coreference resolution. coreference resolution aims at detecting linguistic expressions referring to the same entities in the context of the discourse. the task has been dominated by machine learning approaches since the first learning based coreference resolution system was proposed by @xcite . before @xcite proposed the first end-to-end neural network based coreference resolution system, most of the learningbased systems have been built with hand engineered linguistic features. @xcite use surface linguistic features, such as mention type, the semantic head of a mention, etc., and their combinations to build a classifier to determine if two mentions refer to the same entity. @xcite adopt integer linear programming (ilp) to introduce coreference constraints including centering theory constraints, direct speech constraints and definite noun phrase and exact match constraints in the inference step in order to adapt an existing coreference system trained on the newswire domain to short narrative stories without any retraining. recently, @xcite apply a bert model to coreference resolution and achieve promising results on the ontonotes corpus @xcite and the gap dataset @xcite . different from all coreference systems mentioned above, which rely on supervised learning and on a dataset annotated with coreference links, our work focuses on applying soft linguistic constraints to improve the model's ability of resolving coreferents in an implicit and unsupervised way. similar to the work of venkitasubramanian et al. (2017) that operates on language and vision information, our model uses attention to jointly learn multi-modal representations. ## methodology in this section, we formally describe the visual dialog task @xcite and the approaches we propose. in visual dialog, given an image i, the image caption c and the dialog history until round t1, @xmath0, a 1 ), (q 2 , a 2 ), • • • , (q t-1 , a t-1 )), which is a sequence of question-answer pairs expressed in natural language and grounded in the image, a dialog agent is expected to correctly answer the question at round t by choosing the answer from 100 candidate answers a @xmath1, a 2 t , • • • , a 100 t }. we first introduce the baseline visual dialog model in section 3.1, followed by the detailed explanation of our proposed soft coreference constraints and how we integrate these into the baseline model in section 3.2. the coreference constraints are based on linguistic knowledge, so consequently our method is an example of how to integrate linguistic knowledge into a neural transformer architecture. figure 2 shows the architecture of our proposed model. ## conclusion in this paper, we have built a multi-layer transformer model for the visual dialog task. based on linguistic knowledge and human dialog discourse patterns, we have proposed two soft constraints that effectively improve the model's performance by enhancing its ability of implicitly resolving pronoun coreferences. we have used the visdial v1.0 dataset to evaluate our model. our model obtains new state-of-the-art performance in correctly answering the dialog questions when compared to existing models without pretraining on other visionlanguage datasets. our coreference and qualitative analysis further supports the proposed soft constraints.