They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies (2020)

Abstract: In this paper, we study the grounding skills required to answer spatial questions asked by humans while playing the GuessWhat?! game. We propose a classification for spatial questions dividing them into absolute, relational, and group questions. We build a new answerer model based on the LXMERT multimodal transformer and we compare a baseline with and without visual features of the scene. We are interested in studying how the attention mechanisms of LXMERT are used to answer spatial questions since they require putting attention on more than one region simultaneously and spotting the relation holding among them. We show that our proposed model outperforms the baseline by a large extent (9.70% on spatial questions and 6.27% overall). By analyzing LXMERT errors and its attention mechanisms, we find that our classification helps to gain a better understanding of the skills required to answer different spatial questions.

## introduction visual dialogues are a useful testbed to study how models ground natural language and in particular how they ground spatial language, which is the focus of our analysis. visual dialogues have been the aim of early work on natural language understanding (nlu) @xcite and are now studied by a very active community at the interplay between computer vision and computational linguistics (e.g. @xcite ; @xcite ; @xcite ). recently, important progress has been made on visual dialogue systems thanks to the release of datasets like vis-dial @xcite and guesswhat?! @xcite . the former contains chit-chat conversations about an image whereas the latter is a visual game, hence its dialogues are goal-oriented. in both cases, one agent asks questions and the absolute is it the bus on the left? no relational is the boat next to a car? no group is one of the two in the back? yes we classify them as absolute, relational, and group based on how they many objects are involved and how they are related. the red box marks the object(s) involved in the question, while the green box marks the target of the game. relational and group questions need more than one object, whereas absolute do not. it is interesting to understand where current multimodal nlu models stand with respect to this task: answering questions asked by humans in a goal oriented visual dialogue. our paper addresses this question by evaluating how the oracle model of the guesswhat?! game answers questions asked by humans while playing the game. guesswhat?! is a cooperative game where two players talk in order to identify an object in an image. the player known as the questioner has to guess the target by asking yes/no questions. the other player, the oracle, knows the target object and answers the questions. @xcite show that most of the questions in the dataset are about the entity of the target ("is it a female?") or its location ("is it the first one?"). @xcite show that the baseline model, commonly used for the guesswhat?! task since its introduction in de @xcite , has almost human-like accuracy on the entity questions and a much lower accuracy on questions about attributes. in this paper, we focus on spatial questions and classify them into three groups: absolute, relational, and group questions as illustrated in figure 1 . an unpleasant aspect of the baseline model is that it receives the gold standard entity of the target (that is, the category label, e.g. "giraffe" or "boat") as input. furthermore, it answers questions without seeing either the image or the visual features of the target, but instead it simply relies on the category label of the target and its coordinates. important progress on multimodal encoders has been obtained since the guesswhat?! release; hence, we study the effect of using models that ground the question into the image and do not have access to the gold standard category label of the target. we adapt a multimodal universal encoder, lxmert @xcite , to play the role of the oracle and compare it with the baseline model. it is known that grounding spatial expressions is challenging for neural networks since quite often they require models to put attention on more regions simultaneously and spot the relation holding among them (e.g., the car and the boat in figure 1 , middle). lxmert is a transformer-based neural network and as such it heavily exploits attentionbased mechanisms. in this paper, we run a qualitative analysis of the attention lxmert exhibits for the different types of location questions and run an in-depth error analysis of its results. to sum up, we make the following contributions: the paper proceeds as follows. section 2 reviews previous work on visual question answering and on spatial referring expressions. section 3 presents the models providing information on how we adapt lxmert for the oracle task. section 4 describes the dataset and our classification of spatial questions. in section 5 we compare the accuracy of the models reporting a fine-grained evaluation by question type and zoom into the subset of spatial questions. we further analysed this subset through a manual inspection of lxmert attention and errors in section 6, before drawing our conclusions in section 7. the code of our work is available at: https://github.com/albertotestoni/ unitn_unc_splu2020 . ## related work answering visual questions is a task that has received increasing attention during the last years. interesting exploratory analysis has been carried out to understand visual question answering (vqa) systems which highlight their weaknesses and strengths, e.g. @xcite @xcite . vqa datasets contain both wh-and y/n-questions. but the kind of y/n visual questions the oracle needs to answer are different than those of the vqa datasets: it has to check whether the target has or does not have the questioned property. hence, it has to compare the target's properties with those of the entity the question refers to and answer accordingly. moreover, differently from vqa, the guesswhat?! dataset has been collected in a more naturalistic environment, by letting humans play the games. we adapt lxmert @xcite , a multimodal universal encoder state-of-the-art in vqa, to accomplish the oracle's challenge. after the introduction of the supervised baseline models @xcite , several models have been proposed for the questioner, which are mostly based on reinforcement learning @xcite @xcite @xcite @xcite . for these models, the role of the oracle is even more salient than for models based on supervised or cooperative learning @xcite since they are reinforced to ask those questions the oracle is good at answering. despite this important role of the oracle, no work has been carried out to evaluate and improve it. we aim to fill this gap. @xcite show that guesswhat?! human players ask quite a lot spatial questions. it has been observed that capturing the spatial relation about objects is challenging for neural network models. @xcite argue that convolutional neural network (cnn) do not ground spatial information properly: since they discard location information through the pooling mechanism, their embeddings can only capture rough relative positions of objects within a scene. in line with this claim, @xcite show that linguistic features are more spatially informative than cnn visual features. new multimodal models, like lxmert, start from positional aware embeddings. we therefore study how well they handle the spatial questions asked by guess-what?! players. spatial expressions have been deeply studied within the referring expression generation community. in this area, earlier work @xcite has suggested that, in ordered domains (e.g., a document divided into sections and subsections), referring expressions that include spatial information, even when redundant, lead to a significant reduction in the amount of search that is needed to identify the referent. it has been argued that spatial information reduces the cognitive load (measured by eye tracking) necessary for resolving a referring expression @xcite . this research area @xcite distinguishes between spatial referring expressions that involve another object in the description (e.g. "the rabbit in the hat") from those that do not (e.g. "the rabbit on the left"). the first group of expressions is known as relational, while we shall refer to the second one as absolute. a further distinction is made between referring expressions that are singular (e.g. "the rabbit in the hat") and those that are plural (e.g. "the three rabbits on the table") and refer to a group (see e.g., @xcite ; @xcite ; @xcite ). in this paper, we classify guesswhat?! spatial questions using absolute, relational and group distinctions and examine how lxmert performs for each type of spatial question. we also conduct an error analysis and an attention analysis taking these categories into consideration. recent work by @xcite shows that in current visual dialogue datasets the dialogue history rarely matters. the authors ask crowdsourcers whether they can confidently answer a question by looking at the image and the question, without seeing the dialogue history. in our qualitative analysis we check whether history plays a role for the spatial questions of the guesswhat?! game that lxmert fails to answer. ## models in this section we present the models that we compare. we also explain how we adapted lxmert to the oracle task. the models are trained on successful games. lstm is the baseline model proposed in @xcite . it does not have access to the raw image features. it receives as input embeddings of the target object's category, its spatial coordinates, and one question encoded by a dedicated lstm. these three embeddings are concatenated and fed to a multi-layer perceptron (mlp) that gives an answer (yes or no). v-lstm we enhance the lstm model described above with the visual modality and we remove the information about the target object category. we extract the visual vectors corresponding to the input image and the crop of the target object using a frozen resnet-152 network pre-trained on imagenet @xcite and we pass them through a linear layer and a tanh activation function. we concatenate these scaled representations to the embeddings of the target object's spatial coordinates and the question: the resulting vector is fed to an mlp to obtain the answer, as it happens in the lstm model. lxmert to evaluate the performance of a universal multimodal encoder, we employ lxmert (learning cross-modality encoder representations from transformers) @xcite . it represents an image by the set of position aware object embeddings for the 36 most salient regions detected by faster r-cnn and it processes the text input by position aware randomly initialized word embeddings. we fill the 36th position with the visual features of the target object. both the visual and linguistic representations are processed by a specialized transformer encoder based on selfattention layers; their outputs are then processed by a cross-modality encoder that through a crossattention mechanism generates representations of the single modality (language and visual output) enhanced with the other modality as well as their joint representation (cross-modality output). lxmert uses the special tokens cls and sep; the latter is used to separate sequences and to denote the end of the textual input. lxmert has been pre-trained on five tasks. 1 it has 19 attention layers: 9 and 5 self-attention layers in the language and visual encoders, respectively and 5 cross-attention layers. we process the output corresponding to the cls token. we consider both the pre-trained version (lxmert) and the one trained from scratch (lxmert-s). @xcite where a question can be assigned to more than one attribute type (multiple labels); the single label column reports the number of questions which have been assigned to only one type. ## the dataset the guesswhat?! dataset is composed of more than 150k human-human dialogues containing an average of 5.3 questions in natural language created by turkers playing the game on ms coco images @xcite . humans have succeeded on 85% of the games. not successful games may contain errors made by the human oracle which lead to task failure, we discard questions that belong to human dialogues that were not successful. the remaining set contains around 672k questions which are grounded on about 63k unique images and belong to about 135k dialogues. @xcite propose a classification of the questions based on their focus distinguishing questions which ask about the entity of the target ("is it an animal?" or "is it a dog?") or an attribute of it. a question can focus on just one attribute (e.g., "is it the black dog"? or "is it black?") in which case it is assigned just to one attribute question type (color in the examples) or about more attributes (e.g., "does it have orange pillows on it?") in which case it is assigned to more attribute question types (to both color and spatial information in the example.) we conjecture that the spatial question type includes questions posing different challenges to multimodal models. @xcite divide spatial expressions into relational (e.g. "the cross-modal attention layers. the model behaves similarly to the more complex version trained from scratch. rabbit in the hat"), that specifies the location of the referent of a noun phrase (the target, "rabbit") relative to another object (the landmark, "hat"), and absolute that focus only on the target by providing locative information about it (e.g. "the rabbit on the left"). a third spatial expression that has received attention within the reg community are group referring expressions whose target is a group of entities (e.g. "the three rabbits on the table") or some specific entity of a the group to which the expression refers by ordering them (e.g. "the second rabbit from the left"). we adapt such classification to the guesswhat?! spatial questions and classify them into four types: relational, absolute, group and other. to distinguish these types we have leveraged syntactic and lexical characteristics specific to each. relational questions usually include a prepositional phrase followed by a noun phrase that includes either a pronoun (e.g. "is there a sink directly above it?") or an object word (e.g. "is it the pen behind the laptop?"). absolute spatial questions (e.g. "the one on the left?") instead contain a location word either in the x axis (e.g. right, middle, left), or the y (top, bottom), or the z (e.g. front, back) axis. we also consider absolute those questions that include a spatial adjective in its superlative form (e.g. "the leftmost one?"). finally, we consider group questions those containing a number which may indicate order (e.g. "right to left, is it the first one?") or groups (e.g. "in the back among four women?"). we have automatically annotated spatial questions by identifying nouns, prepositions and number using the part of speech tagger stanza @xcite . when a question is not assigned to any of the three groups, we include it in the "other" category. 3 we tried identifying objects using the entity recognizers included in stanford core nlp @xcite and stanza @xcite but the coverage was not good. in the next section, we will first compare models using the multi-label classification reported in ## qualitative analysis as a first step towards a deeper understanding of lxmert performance, we use a linear logistic regression model for the task of predicting whether a question was answered correctly. in @xcite it has been shown that unsuccessful games contain more objects in the image than successful ones, and that the target size area is smaller. we use these two features as predictor variables together with the length of the question and the turn in which it was asked in the full dialogue. we observe that the number of objects in the image and the question turn play a significant role in predicting the model behaviour. this might be due to the fact that models do not receive the dialogue history as input. below we run an error analysis based on the three spatial sub-type questions described above to check whether indeed this could be a source of error. after the error analysis, we study whether lxmert uses its cross-modal attention differently across these three groups of questions. ## conclusion in this paper we tackle the problem of grounding spatial questions in the guesswhat?! visual dialogue game. we adapt lxmert to play the role of the oracle of the guesswhat?! game reaching an overall accuracy of 82.21%. this result outperforms the widely used baseline model by 6.27%. the gain is even higher for spatial questions, where lxmert outperforms the baseline by 9.70%. in order to perform an in-depth analysis, we classify spatial questions into three sub-types and use this classification to annotate the subset of spatial questions in the guesswhat?! test set. the fine-grained evaluation shows that the hardest spatial questions are the relational and group ones. we perform an in-depth analysis of lxmert cross-modal attention and an qualitative analysis of the errors on each question sub-type. first of all, we find out that lxmert puts attention on more regions when processing relational questions compared to absolute and group questions. secondly, the qualitative analysis highlights the importance of having access to the dialogue history in order to answer some spatial questions. we leave this for future work.