On the role of effective and referring questions in G uess W hat?! (2020)

Abstract: Task success is the standard metric used to evaluate referential visual dialogue systems. In this paper we propose two new metrics that evaluate how each question contributes to the goal. First, we measure how effective each question is by evaluating whether the question discards objects that are not the referent. Second, we define referring questions as those that univocally identify one object in the image. We report the new metrics for human dialogues and for state of the art publicly available models on GuessWhat?!. Regarding our first metric, we find that successful dialogues do not have a higher percentage of effective questions for most models. With respect to the second metric, humans make questions at the end of the dialogue that are referring, confirming their guess before guessing. Human dialogues that use this strategy have a higher task success but models do not seem to learn it.

## introduction guesswhat?! @xcite is a cooperative two-player referential visual dialogue game. one player (the oracle) is assigned a referent object in an image, the other player (the questioner) has to guess the referent by asking yes/no questions. referential visual dialogue has a clear task success metric: whether the questioner is able to correctly identify the referent at the end of the dialogue. the need of going beyond this metric to evaluate the quality of the dialogues has already been observed. so far attention has been put on the linguistic skills of the models @xcite and their dialogue strategies @xcite . but still the models are evaluated without considering how much each question contributes to the goal. we propose two new metrics for evaluating questions. first, a question is effective if it rules out at least one possible distractor @xcite . second, a question is referring if it uniquely identifies one object in the image. figure 1 gives a game played by humans as an example. in the image there are 8 candidate objects: the referent object is the cow marked in green and the distractors are the other 6 cows and the wooden stick. the dialogue is highly effective: 80% of the questions eliminate at least one distractor. the figure shows for each question its answer, how many distractors (#d) are left after each answer and whether the question is effective or not effective. the last question is not effective but it is referring, it uniquely identifies the referent. interestingly, question 2 is also referring but not with respect to the referent, so the dialogue needs to go on. in the next section we review previous work. then, we define the metrics formally and calculate them over the guesswhat?! sota models. finally, we argue that models, differently from humans, do not confirm their guess before guessing. ## previous work despite recent progress in the area of vision and language, recent work @xcite in the navigation task (vln) argues that current research leaves unclear how much of a role language plays in this task. they point out that dominant evaluation metrics have focused on goal completion rather than how each action contributes to the goal. historically, the performance of vln models has been evaluated with respect to the objective of reaching the goal location @xcite . the nature of the path an agent takes, however, is of clear practical importance: it is undesirable for any robotic agent in the physical world to reach the destination by taking a lot of deviation or getting into dangerous zones. jain et al. (2019) propose alternative metrics that evaluate the intermediate human question answer #d effective 1. is it a cow? yes 6 true 2. is it the big cow in the middle? no 5 true 3. a cow on the left? no 3 true 4. in the front? yes 0 true 5. first cow near us on the right? yes 0 false figure 1: human-human dialogue on the guesswhat?! referential task extracted from (de vries et al., 2017). the referent is highlighted in green. #d is the number of distractors remaining after the question is answered. four out of five questions eliminate distractors and, hence, are effective according to our definition. the last question is referring with respect to the intended referent. following this idea of not only focusing on goal completion but on evaluating how much each step contributes to the goal, in this paper we propose two new metrics for referential dialogue. we agree with @xcite that incremental evaluation metrics such as ours should look further back into the dialogue history. we believe that language and vision systems should also be evaluated on aspects such as grammatically, truthfulness, diversity and other aspects as done in previous work @xcite @xcite . in this paper we focus on whether a question is effective and referential considering the dialogue history and the visual context. one of the motivations for referential visual dialogue is to provide robots with the ability to identify objects through dialogue with a human as the robot moves. the task we address in this paper is a simplification. in our setup, the view of the robot is static, it is a picture. for our work we use the guesswhat?! dataset @xcite . recently, @xcite showed that several end-to-end dialogue systems do not take dialogue history into account. in this paper we are particularly interested in the guesswhat?! models that generate questions explicitly modelling the dialogue history @xcite @xcite ). foot_0 ## dataset and evaluation metrics in this section we briefly introduce the dataset and we define the evaluation metrics that we use. ## experiments and results in this section we describe the guesswhat!? sota models for which the code or the test set dialogues have been released and we present our results. ## conclusions we proposed two new metrics for evaluating guesswhat?! dialogues. effectiveness, as we defined it, evaluates whether the question can rule out at least one possible distractor. we consider a question to be effective if it is able to make the reference set smaller both if the question is answered with 'yes' as well as if it is answered with 'no'. we observe that it decreases as dialogues advance and reaches its lowest level in the last turn. we also find that successful dialogues do not have a higher percentage of effective questions. this is surprising, and hints at the fact that there are other strategies to humans seem to use the last turn to confirm their guess before guessing. human dialogues that confirm the guess using a referring questions have a higher task success than those which do not. we plan to explore whether models can learn to confirm their guess before guessing. as future work we plan to refine our referring metric. we have observed that some dialogues do not make explicit the object category in the confirmation. e.g. "the one near us on the right?" in figure 1 . by our definition, this question would not be referring because it is also true for the wooden stick. we believe that our metrics could be heuristics that guide the training of end-to-end models.