The Impact of Answers in Referential Visual Dialog (2021)

Abstract: In the visual dialog task GuessWhat?! two players maintain a dialog in order to identify a secret object in an image. Computationally, this is modeled using a question generation module and a guesser module for the questioner role and an answering model, the Oracle, to answer the generated questions. This raises a question: what’s the risk of having an imperfect oracle model?. Here we present work in progress in the study of the impact of different answering models in human generated questions in GuessWhat?!. We show that having access to better quality answers has a direct impact on the guessing task for human dialog and argue that better answers could help train better question generation models.

## introduction collaborative reference resolution is a task that has attracted a lot of attention in recent years with the introduction of guesswhat?! @xcite . guesswhat?! is a cooperative two-player referential visual dialogue game. one player (the oracle) is assigned an object in an image and the other player (the questioner) has to guess the referent by asking yes/no questions. an example of a dialog in the guesswhat?! dataset can be seen in figure 1 . in this task, much work has been done on the question generation policies @xcite @xcite , the linguistic capabilities of these questioner models @xcite and on improving guessing models @xcite . most of the work on the questioner models was performed employing a simple oracle model to play the guesswhat?! game. this oracle model was too simple and it struggled to answer questions that asked for anything beyond the available annotation information, thus pushing models to produce those type of questions @xcite ; a new sota for the oracle task @xcite , ## previous work guesswhat?! (de vries et al., 2017) is a cooperative guessing game in which two players hold a dialog intended to identify a secret object in a picture. we call this object the target object. the two players have different roles: the questioner has to pose questions and guess the object at the end of the dialog, and the oracle has to answer these questions. the corpus comprises more than 155k dialogs with more than 821k question-answer pairs made across 67k images extracted from the mscoco dataset @xcite . guesswhat?! is a simplification of the collaborative process of referring studied by @xcite . the process of multimodal reference resolution had received attention from the vision and computational linguistics communities @xcite . the task requires both reference resolution capabilities and the ability to ground the language expressions to objects in the real world @xcite . in this task, much work has been done on training question generation policies to perform the questioner role @xcite @xcite @xcite with different levels of task success at guessing the target objects. most of the approaches receive some sort of reward that weights to some extent the task success at the game. being a two player game, the task success will be conditioned by both the questioner performance and the oracle performance. most works use the same oracle model proposed with the guesswhat?! dataset (de vries et al., 2017) foot_1 . we previously showed that the baseline oracle proposed by @xcite does not have the same performance for human and model generated questions, and that performance was linked to the type of question @xcite . most rlbased models would not ask for information other than the type of object and its location, exactly the two manually annotated features that the oracle receives. as a result the grammatical and lexical diversity of the generated questions is poor. following this line, @xcite proposed a more complex oracle model based on the multimodal transformer lxmert @xcite . this model achieved sota for the oracle task and proved to perform better across most question types except for object questions (due to not having access to the gold standard category label for the target). the impact of answers has previously been noted for the visdial task @xcite . @xcite show that a visual dialog model with integration of better answers achieves better performance in the visual dialog challenge 2018. we show the impact the answer has for the guesswhat?! task. ## guesswhat?! task and models in the guesswhat?! game there are two roles: the questioner, that makes questions and guesses the target object at the end of the dialog, and the oracle that answers those questions. at the beginning of a game, the oracle is assigned an target object in the image and the questioner has to pose yes/no questions in order to identify the target. an usual computational modeling for each player divides the questioner role into two components: the question generator and the guesser. in our experiments we make use of a guesser model and oracle models. for the oracle models we used the baseline model @xcite as well as the lxmert based model @xcite . the baseline models (figure 2a ) use a set of features passed through a multilayer perceptron (mlp). we consider a subset of the features according to previous work (de @xcite : the question (q), the spatial information of the target (sp), the target object's category extracted from the mscoco (ca) and the visual features of crop of the target (cr), extracted with a resnet152 @xcite . the lxmert based model (figure 2b ) receives the question, the visual features of 36 regions of the image (the same as in @xcite ) and the crop of the target inserted in the 36th position of the regions. notice that this model has no access to the category's label for the target object. we use the guesser model (figure 3 ) proposed by @xcite . this guesser model adds an encoding of both vision (the image) and language modalities (the full history). a single mlp processes each object's spatial information and category and outputs a score for each object. these concatenated scores combined with the vision and language encoding output the probability of each object being the target used to make the final guess. ## experiments in this section we show the results of our experiments. our experiments were performed using a single 1080ti gpu. we retrained all of our models following the procedure stated in their respective paper using the publicly available source code. for our experiments we take the human dialogs from the test set of guesswhat?!. we keep the human posed questions but change the answers with the ones given by the different oracle models we employed. we stick to the successful games in the test set, as failed and incomplete games tend to contain malformed questions, misunderstandings or are not finished and, thus, guessing is not possible with the information available. we measure the task success of the guesser at guessing the target object in these resulting dialogs. in the lxmert model, despite not having the gold stardard labels for the categories achieves a similar and even higher performance when paired with the guesser model. this shows that the lxmert oracle can be used in settings with no annotation for the objects categories. in figure 4 we see an example of a dialog and the answers given by different oracles. in this example, models other than lxmert miss at least one answers and fail at guessing. this shows that missing a single answer can end up in failure at guessing question human q+sp q+sp+cr q+sp+ca lxmert 1. is it a ship? yes yes no yes yes 2. is it white? yes no no yes yes 3. is it under the plane slightly left? yes yes no no yes status success failure failure failure success figure 4: example different answers for the same human dialog given by the different oracles. human corresponds to the human answers given in the corpus. in this example, the guesser model correctly guesses the target for the human and lxmert oracles, while the game resulted in failure for the rest of the oracle models. the target object, as seen for the q+sp+ca and q+sp oracles. oracle ians avg ians/failure q+sp 31.73% 2.08 q+sp+cr 25.60% 1.78 q+sp+ca 22.03% 1.66 lxmert 16.05% 1.27 following this we compute the percentage of incorrect answers (ians) and the average amount of them per failed game (avg ians/failure) for each model. in ## conclusions in this paper we described work in progress on studying the impact of having access to better answers in guesswhat?!. dialogs with better answer quality had a higher task success when sent to an automatic guesser. the task success when using the widely used q+sp+ca and the lxmert oracles are comparable although lxmert does not require the object manual annotations. task success drops when the gold standard category label for the target is not a feature. this suggest that the mlp oracles rely strongly on the manual annotations. the next step will be to investigate the impact of the answer quality on the quality of the generated questions. in order to do so we will perform a similar analysis for the different questioner and guesser models proposed in the literature. we hypothesize that this could lead the question generation policies to have richer linguistic capabilities and to learn better strategies for identifying the target object.