Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History (2021)

Abstract: In this paper, we define and evaluate a methodology for extracting history-dependent spatial questions from visual dialogues. We say that a question is history-dependent if it requires (parts of) its dialogue history to be interpreted. We argue that some kinds of visual questions define a context upon which a follow-up spatial question relies. We call the question that restricts the context: trigger, and we call the spatial question that requires the trigger question to be answered: zoomer. We automatically extract different trigger and zoomer pairs based on the visual property that the questions rely on (e.g. color, number). We manually annotate the automatically extracted trigger and zoomer pairs to verify which zoomers require their trigger. We implement a simple baseline architecture based on a SOTA multimodal encoder. Our results reveal that there is much room for improvement for answering history-dependent questions.

## introduction the development of multimodal conversation agents is a long standing challenge (e.g. @xcite ). in recent years, much has been achieved on the challenge of visual question answering (vqa) (e.g. @xcite .) the rapid advancements have brought researchers to further increase the difficulty of the task by proposing visual dialogue datasets (e.g. @xcite ) suitable to train multimodal dialogue systems. with this switch from vqa to visual dialogue, the challenge has increased in difficulty. first of all, while vqa involves only understanding the multimodal input (image and question), visual dialogues also require visual question generation and the acquisition of a dialogue strategy. moreover, while vqa involves visual grounding of the question to be answered, visual dialogues require grounding the question against questioner oracle q1. is it a fruit? yes q2. is it in the foreground? no q3. are there two of them on the branch? yes q4. is it the top one? yes figure 1: through the dialogue the focus shifts from all the mandarins to just one. to answer q4 (the zoomer), "top" needs to be interpreted relatively to the group of two mandarins identified by q3 (the trigger). work carried out on modeling the role of dialogue history in visual dialogue @xcite has used the chit-chat dialogues of vis-dial @xcite as a case study. however, it has been shown that in this dataset the role of grounding the question on the dialogue history is limited: models that take history into account do better, but the dataset contains a small percentage of questions that require dialogue history to be interpreted correctly. based on these findings, agarwal et al point out the need for data which captures dialogue history dependence. our work is a contribution to this data collection challenge. we aim to identify fuqs which require (part of) the dialogue history to be interpreted. @xcite claims that goal-oriented settings will contain more dialogue phenomena. following this claim, we run our analysis on @xcite , a multimodal dataset in which the goal of the dialogues is to identify a referent in an image. in referential dialogues, the questions aim to collect information so as to narrow the set of potential candidates and univocally identify the referent among them. interestingly, in referential multimodal game, this progressive refinement happens both through the language and visual contexts by incrementally zooming the joint attention to the conjectured referent. for instance in figure 1 , q1 focuses on the full image and all objects are potential candidates. as the dialogue proceeds the attention is moved on the mandarins (q1), then on those two mandarins in background (q2) and finally on the mandarin on the top of the group of mandarins in the lower branch (q3 and q4). in this view, the data collection challenge launched in @xcite ) can be rephrased by looking for a method to extract fuqs which require to zoom on a specific region of the image by narrowing the set of entities on which the dialogue focuses. we claim that fuqs requiring multimodal grounding can be extracted by identifying patterns of trigger-zoomer questions. by manual inspection of the human dialogues, we have observed that often, after a positively answered question, the questioner tries to narrow down the choice by asking further details that discriminate the candidate. this happens in particular, when first a question (the trigger) identifies a group of objects that share some property and then the fuq (the zoomer) focuses on one or more of the members of the identified referential set. in most cases, the zoomer requires the dialogue history to be answered. for instance, in figure 1 , the positively answered q3 acts as a trigger which identifies the group of oranges under discussion and q4 zooms on one of those. notice that the question q4 would be answered incorrectly if answered without considering q3 and q2 because the referent is not at the top of the picture. in this paper, we investigate the role of spatial questions in the identification of such patterns and focus on the evaluation of the oracle player of the guess-what?! game. we show that the method we propose facilitates data collection of follow-up questions that need to be grounded on the visual and dialogue context to be answered correctly or at least with higher confidence. the dataset is publicly available at https://github.com/tianaidong/ 2021splu-robonlp-vispa for future model developments and evaluations. most of the work on the guesswhat?! game has focused attention on the questioner player; as a consequence, the issue of dialogue history needed by the oracle has never been considered. since the first baseline model @xcite , the oracle receives just the question without the previous turns. furthermore, this baseline model is blind: it takes the question, the target's category and its location as inputs. this simple model has been widely used as the oracle agent by all work on the questioner (eg. @xcite @xcite .) @xcite compared the lstm baseline with a visually grounded lstm (v-lstm) and with an adaptation of lxmert @xcite . they show lxmert based oracle improves over the baseline achieving a new sota for the guess-what?! oracle. yet the model does not use dialog history as an input. we evaluate lstm, v-lstm and lxmert against our dataset of contextdependent questions. @xcite argues that although complex models that encode history for visual dialogs have been proposed @xcite , such work has not demonstrated that history matters for visual dialogs. @xcite propose and apply a new methodology for evaluating history dependence of questions in visual dialog. they show crowdsourcers a question with its image without the dialog history and ask the crowdsourcer "would you be able to answer this question by looking at the image only or you need more information from the previous conversation?". however, it could happen that workers could be confident in answering the question just by looking at the image, but that they would give a different answer if the dialogue history is provided. this difference is crucial for studying context-dependent questions. in this paper, we proposed a new methodology for detecting history-dependent visual questions. ## dataset we aim to identify follow-up questions (fuqs) that need the previous turn to be answered correctly or at least with higher confidence. we claim that fuqs which zoom in a specific region of the image to identify an object (or a set of objects) in it satisfy this request. this might hold in particular when the region contains more objects of the same category (e.g., more instances of mandarins, as in figure 1 ) and the question refers to one (or more) member(s) of such a group. moreover, we conjecture that most of such questions might also need to be visually grounded since the answer to it could change if the specific visual region they refer to is not properly identified and the question is mistakenly grounded over the full image. these challenging questions that zoom into a group are usually triggered by a question that refers to the whole group, the latter is identified by its location, the number or the color of its members. for instance, in figure 1 the question that zooms on the target object of the game, "is the top one?', is triggered by the previous question that identifies the group itself by referring to the number of its members "are there two of them on the branch?". interestingly, the zoomer question would be answered incorrectly without the previous turn since the target is at the top of the zoomed region and not on the top of the full image. we focus on games in the test set in which there are more candidates of the same category of the target; we obtain 13,024 unique games containing 57,241 questions. we refer to these questions as the full test set. @xcite has classified guesswhat?! questions into entity and attribute questions, the latter are subdivided into spatial, color, action, size, texture and shape. @xcite further divided the spatial questions into group, absolute and relational questions. we build on these classifications to extract trigger and zoomer pairs. we see group and color questions as potential triggers for collecting history-dependent questions: for instance, group questions that contain explicit numbers indicate groups (e.g., "one of the three oranges?" refers to a group containing 3 members) and color questions might identify a group of objects which differ with respect to the color (e.g., "is it blue?" may refer to a group of objects one of which is blue). for the zoomer questions, we consider group and absolute questions. absolute questions are those spatial questions that contain an absolute location adjective (e.g., "is it in the middle?" contains "middle"). other types of questions, such as size ("is it one of the big bottles?" which contains "big") and shape ("is it kind of round?" which contains "round") could be used as triggers and zoomers as well. in this paper we do not use them because they are not frequent in the guesswhat?! dataset and a preliminary analysis showed we would not extract sufficient trigger zoomer pairs through them. using the automatic annotation of @xcite , which is based on keyword matching, we extract group and absolute questions, 4342 and 11,743, respectively. moreover, we extract the dialogues containing context-dependent group questions using the following patterns: a positively answered group question followed by another group question (group-group) and a positively answered color question followed by a group question (color-group); we obtained 364 and 145 pairs, respectively; and similarly for absolute questions obtaining 919 context-dependent absolute questions (530 from the group-absolute and 389 from the color-absolute patterns). we randomly retrieve 200 samples for each subset foot_0 and manually checked them. we filtered out those pairs in which the zoomer question could be correctly interpreted without the dialogue history. we also removed samples that were noisy (the image was blurry or the target was too small, the question was not clear, etc). each datapoint was annotated by two annotators (the four authors), and we maintained only those on which there was an agreement between the two annotators. after this filtering, we obtained in total 271 contextdependent questions manually checked: 164 group questions (103 group-group and 61 color-group) and 107 absolute questions, the latter are all from the group-absolute pattern. 2 we will refer to the set of visually grounded spatial questions that are context-dependent as vispa. to gain a better understanding of the linguistic features of our dataset, we collect the statistics of question length, nouns and function words (prepositions pronouns, determiners, conjunctions, auxiliaries) for questions in each subset. 3 as we can see in ## experiments we evaluated the models described above when receiving just the question or the question and the previous qa turn, we refer to the latter setting by marking the model names by -dh. we run each model three times (seed: 1, 50 and 100) and report their average together with the significance test results about the difference across runs. we claim the fuqs identified through the triggerzoomer patterns need (at least) the previous turn to be answered properly or at least with higher confidence, this need should be even stronger for the manually filtered subsets. when looking at the context-dependent questions, the standard-deviation among the accuracies reached by the three runs is rather high, hence in order to understand its effect on the comparison between models when receiving just the question and the question together with the previous turn, we have run a statistical significance paired t-test (following the suggestions in @xcite .) the result shows that the difference between the two settings is never significant, except for lstm/lstm-dh on the absolute questions (p-@xmath0.05). this shows that model performance is rather unstable and hence the selection of the binary answer is not properly grounded. this instability is not due to the size of the set: we have computed accuracy of the three runs of lxmert on subsets of 500 and 100 randomly chosen questions and obtained a very low standard deviation. since the questions we have accurately selected are context-dependent, ideally a model should increase confidence in its answers when receiving the context (we simplify by giving just the the previquestioner oracle q2. one of the 3 planes in front? yes q3. is the one in the middle? yes figure 3: context-dependent absolute questions: group-absolute ous turn). to verify this hypothesis, we computed the confidence of lxmert/lxmert-dh (see ## discussion and conclusions visual dialogues are an interesting challenge because of the interplay between the language and visual modality. when focusing on answering visually grounded questions in dialogues, the main challenge they pose in addition to visual question answering is the need of grounding the question against the dialogue history. in our work we define and evaluate a methodology for extracting visually grounded history-dependent spatial questions from visual dialogues. our methodology does not capture all history dependencies in the dataset but it assures that those pairs that are identified are indeed historydependent. the "trigger-zoomer" methodology we propose is evaluated here on the guesswhat?! dataset. one possible question is how generic and applicable this model is in longer and open-world dialogues. we think that this method can be extended to longer dialogues by making the "trigger-zoomer" recursive. moreover, it could be extended to datasets that not only contain questions, but also other forms of language. as far as a "trigger" affects a zoomer question by requiring the dynamic change of the multimodal attention to properly interpret it, the trigger can take any form. for instance, the trigger could be provided in the form of a caption referring to a specific region of an image. we believe that the "trigger-zoomer" methodology would be applicable to all open-word subdialogues that focus on reference resolution. reference resolution is a frequent task in dialogue which takes up a large part of the turns in domains that are complex or need search. see for instance da silva rocha and paraboni (2020). we release both the automatically extracted question pairs as well as the subset of such questions which have been manually verified for context dependence. some of these questions cannot be answered correctly without the previous trigger turn or at least confidence in answering them should be higher when the previous turns are provided. we evaluate the simple oracle models proposed so far in the literature and show that the architecture does not profit from the previous turn as it should. we pose the problem of interpreting follow-up questions as an open problem for the community.