Looking for Confirmations: An Effective and Human-Like Visual Dialogue Strategy (2021)

Abstract: Generating goal-oriented questions in Visual Dialogue tasks is a challenging and longstanding problem. State-Of-The-Art systems are shown to generate questions that, although grammatically correct, often lack an effective strategy and sound unnatural to humans. Inspired by the cognitive literature on information search and cross-situational word learning, we design Confirm-it, a model based on a beam search re-ranking algorithm that guides an effective goal-oriented strategy by asking questions that confirm the model’s conjecture about the referent. We take the GuessWhat?! game as a case-study. We show that dialogues generated by Confirm-it are more natural and effective than beam search decoding without re-ranking.

## introduction quite important progress has been made on multimodal conversational systems thanks to the introduction of the encoder-decoder framework @xcite . the success of these systems can be measured by evaluating them on task-oriented referential games. despite the high task-success achieved and the apparent linguistic well-formedness of the single questions, the quality of the generated dialogues, according to surfacelevel features, have been shown to be poor; this holds for systems based on both greedy and beam search (e.g. @xcite ; @xcite ; @xcite ). @xcite found that when taking these surface-level features as a proxy of linguistic quality, the latter does not correlate with task success, and the authors point to the importance of studying deeper features of dialogue structures. we aim to develop a multimodal model able to generate dialogues that resemble human dialogue strategies. cognitive studies show that humans do not always act as "rational" agents. when referring to objects, they tend to be overspecific and prefer properties irrespectively of their utility for identifying the referent @xcite ; when searching for information or when learning a language, they tend to follow confirmation-driven strategies. modelling such behaviour in language learning, @xcite and @xcite propose a procedure in which a single hypothesized word-referent pair is maintained across learning instances, and it is abandoned only if the subsequent instance fails to confirm the pairing. inspired by these theories, we propose a model, confirm-it, which generates questions driven by the agent's confirmation bias. take the example of a referential guessing game in which an agent has to ask questions to guess an object in a given image. confirm-it will ask questions that reinforce its beliefs about which is the target object, till proven otherwise. for instance, in figure 1 , after learning that the target is a living entity (turn 1), the agent conjectures the target is the dog on the right of the picture (though in principle, it could have been any of the candidates). hence, the decoder generates the question that would let it confirm such belief, "is it a dog?". if its expectations are not met (viz., it receives a negative answer to such question -turn 2b), it moves its attention to another candidate object. we do not claim that our choice represents the optimal strategy to play the game, but we believe that it makes the generated dialogue more human-like. to evaluate this strategy, we take as a test-bed guesswhat?! @xcite , a two-player game between a questioner that has to guess the target, and an oracle (called "external oracle" in the following) who is aware of the target. the widely used architecture of the questioner, gdse, jointly trains a question generator (qgen) and a guesser @xcite . we augment this architecture with a module that simulates an internal oracle. being an "internal" oracle, at test time this agent does not know what the target object is: while at training time it learns to answer questions by receiving the gold standard datapoint (the question, the actual target, and the human answer), at test time it assumes the target is the candidate object to which the guesser assigns the highest probability. hence, the three modules of the questioner straightforwardly cooperate one another. the internal oracle guides the qgen to ask questions that reinforce the guesser's beliefs. concretely, at training time, through supervised learning (sl) the qgen learns to ask human-like questions turn-by-turn, the internal oracle to answer them, and the guesser to guess the target object once the dialogue ends. at test time, we implement a beam search re-ranking algorithm that simulates the single-conjecture learning strategy used by humans: among the questions the qgen generates via beam search, the algorithm promotes the questions whose answer (obtained via the internal oracle that receives the candidate with the highest probability as the target) increases the most the model's confidence in its hypothesis about the target. we run both quantitative and qualitative analyses, and evaluate the effectiveness of the dialogue strategy by asking human annotators to guess the target object given the dialogues generated by confirm-it. we compare results giving the dialogue generated by confirm-it when using the re-ranking algorithm and when generating the question proposed by the plain beam search. we show that the task accuracy of both the conversational agent and human subjects increases when receiving the dialogues generated by the confirm-it re-ranking algorithm. ## related work for open-ended language generation, @xcite claim that decoding strategies that op-timize for output with high probability (like beam search) lead to highly deteriorated texts, since the highest scores are often assigned to generic, incoherent, and repetitive sequences. several works propose reranking strategies on the set of hypotheses produced by the beam search following different criteria @xcite @xcite @xcite to improve both the performance on a given task and the quality of the output. in this work, we present a cognitivelyinspired reranking technique for a visual dialogue questioner agent. in visual dialogue systems, the quality of the output has been improved mainly by aiming at reducing repetitions in the output. this goal has been achieved through reinforcement learning by adding auxiliary objective functions @xcite , intermediate rewards @xcite , regularized information gain techniques @xcite , or intermediate probabilities with an attention mechanism @xcite . different from these works, we do not use the reinforcement learning paradigm and, instead of focusing on improving surface-level features, we indirectly operate on the dialogue structure. @xcite studied the way children and young adults search for information while asking yes-no questions given a set of candidate hypotheses. the authors found that when prior knowledge favours some hypotheses over others, participants asked more hypothesis-scanning questions (i.e., questions that are tentative solutions, with a specific hypothesis that is directly tested). this is in line with the observation in @xcite that humans phrase questions to receive an affirmative answer that supports their theory, and with the broader finding in wason (1960) that they tend to select the information that is in accord with their prior beliefs. inspired by these studies, we propose a new dialogue strategy for playing referential guessing games by exploiting the probabilities assigned by the guesser module to different candidate objects. ## task and dataset guesswhat?! @xcite is an asymmetric game involving two human participants who see a real-world image from mscoco @xcite . one of the participants (the oracle) is secretly assigned a target object in the image, while the other participant (the questioner) has to guess it by asking binary (yes/no) questions to the oracle. the guesswhat?! dataset consists of more than 150k human-human english dialogues containing on average 5.3 questions per dialogue. ## model and re-ranking strategy our model, confirm-it, builds on gdse @xcite . in the latter, the hidden state representation produced by a multimodal encoder is used to jointly train the question generator (qgen) and the guesser module. the image is encoded with a resnet-152 network @xcite and the dialogue history is encoded via an lstm network. qgen uses greedy search to generate questions. to this multi-tasking setting, confirm-it adds an internal oracle trained to answer the question at each turn. moreover, it relies on beam search and, at inference time, it goes through a re-ranking phase which simulates the single-conjecture learning strategy. the model architecture is provided in the supplementary material (sm) and the algorithm is spelled out below. algorithm 1 the confirm-it algorithm require: history h, beam size b, max turns t , image i, distractors d 1:n -1 , target o t , require: candidates c 1:n ← d 1:n -1 + o t require: internal oracle io require: target-aware external oracle 1: for @xmath0: t do 2: oracle provides an answer a to q 10: h ← h + q + a 11: end for algorithm 1 describes the beam search reranking algorithm used by confirm-it to promote the generation of an effective dialogue strategy. given an image, a set of candidate objects, a target object o t , and a beam size of b, at each dialogue turn the model predicts a probability distribution over the set of candidate objects given the current dialogue history. the candidate that receives the highest probability is considered the model's hy- decoding strategy accuracy 5q confirm-it 51.38 beam search 47.03 random re-rank 46.29 greedy search 46.58 none of the features of our case-study are crucial for the method to be applied to other tasks, e.g. it does need the questions to be polar, it does not need the questions to be visually grounded, it does not need the dialogue to be asymmetrical. implementation details foot_0 for the multi-task training, we adopt the modulo-n training proposed in @xcite , i.e. we train the oracle and guesser modules every n (=7) epochs of the qgen. at inference time, we use a beam size of 3 and let the model generate dialogues of 5 turns. ## results we study to what extent the re-ranking phase lets the model generate more effective and more natural dialogues. to this end, we evaluate the confirm-it task-accuracy with and without the ranking phase foot_1 and report qualitative analyses of the dialogues. when the model undergoes the re-ranking phase, confirm-it accuracy has an increase of +4.35% with respect to what it achieves when it outputs the question selected by the plain beam search, and an increase of +4.8% against greedy search. note that, instead, randomly re-ranking the set of questions lowers the performance. this result shows that confirmation-driven strategies help generate more effective dialogues. interestingly, our re-ranking method does not require additional training compared to the sl paradigm. ## discussion and conclusion in this paper, we propose confirm-it, a multimodal conversational model based on a decoding strategy inspired by cognitive studies of human behavior. we show that, through the proposed beam search rereanking algorithm, our model generates dialogues that are more effective (based on task-accuracy) and more natural (based on the dialogues features discussed above). we believe further improvement could be obtained by increasing the performance of every single module. moreover, the structure of the generated dialogues remains to be analysed, total number of objects mentioned. chair-i results: 18.28 (beam search), 15.02 (confirm-it), 4.11 (human dialogues). and we agree with van der lee et al. ( 2021 ) that a proper evaluation should involve humans. in future work, our method can be easily extended to other task-oriented dialogue tasks which involve a conversational agent as far as it has a module that generates questions and a module that performs a classification task. depending on the task at hand, different ways to take intermediate probabilities into account can be designed, but the core idea of the method would not change. peter c. wason. 1960. "on the failure to eliminate hypotheses in a conceptual task. quarterly journal of experimental psychology, 12(3):129-140. sina zarrieß and david schlangen. 2018. decoding strategies for neural referring expression generation. in proceedings of the 11th international conference on natural language generation, tilburg university, the netherlands, november 5-8, 2018, pages 503-512. association for computational linguistics. junjie zhang, qi wu, chunhua shen, jian zhang, jianfeng lu, and anton van den hengel. 2018. goaloriented visual question generation via intermediate rewards. in computer vision -eccv 2018 -15th european conference, munich, germany, september 8-14, 2018, proceedings, part v, volume 11209 of lecture notes in computer science, pages 189-204. springer.