Region under D iscussion for visual dialog (2021)

Abstract: Visual Dialog is assumed to require the dialog history to generate correct responses during a dialog. However, it is not clear from previous work how dialog history is needed for visual dialog. In this paper we define what it means for a visual question to require dialog history and we release a subset of the Guesswhat?! questions for which their dialog history completely changes their responses. We propose a novel interpretable representation that visually grounds dialog history: the Region under Discussion. It constrains the image’s spatial features according to a semantic representation of the history inspired by the information structure notion of Question under Discussion.We evaluate the architecture on task-specific multimodal models and the visual transformer model LXMERT.

## introduction visual dialog (vd) is a task that combines natural language understanding grounded in vision with dialog. being visual, vd is closely related to the area of visual question answering (vqa). on vqa, important progress has been obtained recently with models that connect vision and language and are pre-trained on a variety of tasks @xcite . arguably, less progress has been made on the dialog part of vd, which is the topic of this paper. currently, the two most popular datasets for visual dialog are visdial @xcite and @xcite . the former contains chit-chat conversations about an image whereas the latter contains dialogs about a visual game whose goal is reference resolution, hence its dialogs are task-oriented. reference resolution is a fundamental task in situated dialog @xcite @xcite . questions in reference resolution can be classified as intrinsic of the target ("it is a car?") or relative to the context ("on the left?") @xcite . visual dialog is assumed to require the dialog history to generate correct responses. however, it is not clear from previous work how dialog history is used for vd @xcite . in this paper we define history dependence in terms of a representation that is interpretable as a region of the visual common ground shared between dialog participants @xcite . this representation, which we call region under discussion (rud), is inspired by the pragmatic theory of question under discussion (qud). qud @xcite @xcite is a somewhat overlooked but conceptually fruitful theory for spelling out the connection between the information structure of a sentence or question and the discourse or dialog in which the utterance occurs. in this paper we define rud and use it to connect a question to its visual dialog history; we make the following contributions: 1 • we define what it means for a visual question to require dialog history considering intrinsic and relative visual properties. following @xcite we define dialog common ground to be the commitments that the dialog partners have agreed upon during the dialog. an important part of the common ground is the question under discussion (qud) @xcite . qud is an analytic tool that has become popular among linguists and language philosophers as a way to characterize how a sentence fits in its context @xcite . the idea is that each sentence in discourse is interpreted with respect to a qud. the qud is defined by the dialog or discourse history. the linguistic form and the interpretation of an utterance, in turn, may depend on the qud that provides the constraints that define the utterance's context. similarly, we define a region under discussion (rud) for visual dialog as a representation of the constraints that the dialog history establishes. the interpretation of a question depends on its rud. figure 1 shows a dialog from the guesswhat?! visual dialog dataset @xcite . guess-what?! is a cooperative reference resolution game: two players attempt to identify an object in an image. the questioner does not know the target object and has to find it by asking questions; the oracle knows the target and provides yes/no answers. for each question in the dialog, its dialog history is defined as the previous questions together with their answers @xcite . in the figure, the target is highlighted in green. the baseline oracle model proposed by de vries et al. correctly answers the first four questions, failing only in question number 5 with a no answer. this question does not look particularly difficult. so, why did it create a problem? because question 5 is the only question for which the dialog history modifies the response. all the other questions can be answered correctly just by looking at the image and ignoring what was said before. that is, questions 1 to 4 are vqa turns because they do not need the dialog history. 2 if we answer question 5 is it on the left? ignoring the dialog history the correct answer is no, because the target is clearly to the right of the picture, not to the left. the rud for this question, depicted in blue in the figure, modifies the response. in this work, we model in the rud the constraints that are related to intrinsic properties of the target that have been previously agreed upon 2 we invite the reader to try it: just ignore the dialog history and answer the questions by only looking at the image. ## methodology in this section we describe the dataset and we show how we annotate a subset of questions whose dialog history completely change their responses. we then explain how we build a semantic history for each dialog in order to construct a rud and how we extend oracle models with ruds. ## results and discussion in this section we first report the empirical results of our experiments, then we argue that rud summarizes history in a visually interpretable way through a qualitative analysis. finally we discuss the limitations of our implementation of rud. we performed our experiments with the previously proposed models for the oracle task. we implement both of our models as three-way classifiers using mlps and a cross-entropy loss, accordingly with the relevant literature. for the qcs baseline, we follow de @xcite and use a two layer mlp with relu non-linearities (1024-relu-128) while for the lxmert-based oracle we use a simpler setup with just one layer on top of the cross-modality output of lxmert. our cmo implementation is based on the pre-trained lxmert model from the transformers library @xcite . visual features are the same as in @xcite . we leave the rest of the details of our experiments in the appendix a. ## previous work visual dialog played a prominent role in early work on natural language understanding @xcite and is now the focus of an active community investigating the interplay between computer vision and computational linguistics @xcite . on the guesswhat?! task, most previous research has focused on the questioner @xcite @xcite . recent work suggests that the performance of the oracle agent used by most work @xcite is quite different for types of questions @xcite . questioners that rely on the oracle learn to prefer to ask only those questions that the oracle can answer reliably. this has an impact on the type and linguistic variety of the generated questions, reducing the guesswhat?! task to a simpler linguistic task @xcite . @xcite models the process of finding referring expressions as a collaborative process in which the speakers repair, expand on, or replace the noun phrase in an iterative process until they reach a version they mutually accept. this process is explicitly performed in a guesswhat?! dialog although the role of the oracle is simplified. the oracle model proposed by @xcite is implemented with an mlp (as we described in section 3). they showed that their best performing model was the one that takes the question, the target's category and its location as inputs. this has a major limitation: the model is blind and cannot see the image. this proposed model is widely used as the oracle agent for all of the following research on the questioner. @xcite proposed an adaptation of lxmert @xcite to improve on the previous oracle, achieving a new sota for the guesswhat?! oracle without using dialog history as an input. this work showed various improvements in different types of questions, mainly on questions regarding location and other attributes and a little decrease in performance on object or super category questions due to not receiving the gold standard object category as input from the dataset. their qualitative error analysis suggests that spatial questions are harder because they require history in order to be answered correctly in context. @xcite argues that although complex models that encode history for visual dialog have been proposed @xcite , such work has not demonstrated that history indeed matters for visual dialog. @xcite propose and apply a new methodology for evaluating history dependence of questions in visual dialog. they show crowdsourcers a question with its image without the dialog history and ask the crowdsourcer "would you be able to answer this question by looking at the image only or you need more information from the previous conversation?". but saying i can confidently tell the correct answer just by looking at the image is not the same as answering it in the same way that one would by looking at the previous conversation (remember the example in section 2). most questions are answerable no matter where they appear in a dialog because the answerer accommodates. our method differs in that our's has the advantage of getting history dependent questions that are not evident at first glance (such as "is it on the left?" in figure 1 ). we found a similar percent of questions in the gw dataset that are history dependent, as agarwal et al. did on visdial (12% vs. 11%). this may result in current dialogue models not learning history dependence since current mainstream vision and dialog datasets lack a significant amount of history dependency. dialog history has two characteristics that makes it difficult for current machine learning methods: not only it introduces variability with different histories for the same question, history dependence may also not be lexicalized, as in is it on the left? in figure 1 . history dependency is easier to spot when it is lexicalized with explicit pronouns (e.g. him in 'is it close to him?') or through noticeable ellipsis (e.g. a missing noun such as cars in 'are there two together?'). however, as we see in figure 3 , pronouns in task-oriented vd frequently are not anaphoric to the dialog history but to the image (e.g. the pronoun it in is it a person? is anaphoric to the target). information structure theory @xcite and, in particular, qud @xcite @xcite provide a framework for defining context dependence beyond pronouns and syntactic ellipsis. ## conclusions we proposed a novel interpretable representation for visual dialog history: region under discussion (rud). it constrains the image spatial features according to a semantic representation of the history inspired in the information structure notion of qud. we evaluated our method on models for the oracle task in the guesswhat?! dataset. our results show that our implementation of rud leads to improvements in performance on history dependent questions. we release a manually annotated subset of such questions. our experiments confirm that intrinsic properties do not benefit from dialog management whereas questions that ask for properties relative to the context see an improvement with it. interestingly, only a low percentage of questions (12%) are indeed history dependent in the guesswhat?! dataset. however, a single error in a 10 turns gw dialog may cause the identification of the wrong referent, rendering the task unsuccessful. we agree with de vries et al. (2020) that the simplified yes-no nature of this task allows us to focus on an interesting playground for working on conceptual advances in representation methods for dialog history. the guesswhat?! task is ill-suited for incremental research, as it is unclear how small improvements will find their way to real applications. our contribution is not incremental. our paper makes a theoretical contribution by defining the new concept of region under discussion and linking it with the concept of question under discussion in dialog. based on this theoretical contribution it proposes an interpretable, simple and extensible method for representing dialog history. this work only adjusts the rud to reflect the intrinsic properties of the target entity, not other attributes (color, shape, etc.) and spatial restrictions ("is it among the four in the back?"). including other types of relations in the generation of the ruds is a promising avenue for future research. in this regard, we are considering the following approaches: 1) rud generation from scene graphs (a sg is a graphical representation of an image that encodes objects as nodes and pairwise relations as edges), and 2) learning rud predictors from dialog data end-to-end. in both cases, we need a large and representative training set (sg/rud annotated for each turn on each dialog) and such data is hard and expensive to gather. a possible solution in this case is to explore weakly supervised strategies, where the sg/rud is treated as a latent variable. we think that these contributions can be of use for the questioner model, potentially helping questioners learn dialog strategies instead of solving dialog tasks through visual question answering. quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics. tianhao yang, zheng-jun zha, and hanwang zhang. 2019. making history matter: history-advantage sequence training for visual dialog. in 2019 ieee/cvf international conference on computer vision, iccv 2019, seoul, korea (south), october 27 -november 2, 2019, pages 2561-2569. ieee.