“ I ’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in G uess W hat?! (2021)

Abstract: Natural language generation systems have witnessed important progress in the last years, but they are shown to generate tokens that are unrelated to the source input. This problem affects computational models in many NLP tasks, and it is particularly unpleasant in multimodal systems. In this work, we assess the rate of object hallucination in multimodal conversational agents playing the GuessWhat?! referential game. Better visual processing has been shown to mitigate this issue in image captioning; hence, we adapt to the GuessWhat?! task the best visual processing models at disposal, and propose two new models to play the Questioner agent. We show that the new models generate few hallucinations compared to other renowned models available in the literature. Moreover, their hallucinations are less severe (affect task-accuracy less) and are more human-like. We also analyse where hallucinations tend to occur more often through the dialogue: hallucinations are less frequent in earlier turns, cause a cascade hallucination effect, and are often preceded by negative answers, which have been shown to be harder to ground.

## introduction recent years have witnessed important progress in the quality of the output generated by deep neural network architectures. although it is not easy to evaluate the output of natural language generation systems, some features clearly deteriorate their value, making these systems hardly employable in real-world scenarios. crucially, state-of-the-art models are shown to generate words that are not consistent with the source inputs. this issue is generally referred to as hallucination. this phenomenon applies to different nlp tasks and neural architectures. it has been explored in summarization @xcite , machine translation @xcite is it a dog ? no is it a chair ? no is it a fridge ? no is it a cup ? yes on the right? yes is it a person ? no is it a skateboard ? no is it a car ? yes is it white ? no is it green ? no figure 1: hallucinations generated by the gdse model playing guesswhat?!. note that the dialogue on the right also contains a question referring to an attribute (green) that is not related to the source image. in this paper, however, we focus only on entity hallucination. our results confirm that hallucination heavily affects the output of generative models playing guesswhat?!, but pre-trained transformers (used both as encoder and decoder) show a consistent improvement in this respect. moreover, our results reveal that the rate of object hallucination increases across the dialogue turns. hallucinations frequently appear in consecutive turns and are more likely to occur after negative answers. finally, we carry out an in-depth analysis in dialogues produced by human annotators. the main contributions of this paper can be summarized as follows: ## models to allow for a fair comparison of different questioner models, we use the same oracle and guesser models in all our experiments. following de vries et al. (2017), we employ distinct computational models for each of the three key tasks: answering questions (oracle), guessing the target (guesser), and asking questions (questioner). ## conclusion entity hallucination is one of the major problems that affect natural language generation systems in many nlp tasks, from machine translation to image captioning. generating tokens that are not related to the source data compromises the possibility to use these systems in real-world scenarios. in this work, we explore to what extent this problem affects multimodal conversation agents playing the guesswhat?! referential guessing game. we adapt two multimodal transformer-based models to play the guesswhat?! questioner agent based on multimodal transformers architectures (lxmert-gdse and vlp), and we compare their output with the widely used gdse model @xcite and the baseline model in @xcite . we adapt the chair metric proposed in @xcite for image captioning to assess the models' rate of object hallucination. our analysis confirms recent findings about the inadequacy of the task success in the guessing game to serve as a good proxy of the quality of the generated dialogues. while all the models perform similarly in the guesswhat?! game, the dialogues they generate differ dramatically. vlp and lxmert-gdse generate less than half of the hallucinations compared to gdse and the baseline model, confirming the crucial role played by a strong visual processing to reduce hallucinations. the results of our indepth analysis support the hypothesis in @xcite that hallucinations tend to appear at the end of the sequence. moreover, our results reveal that, in most cases, hallucinated tokens follow polar questions answered negatively. we conjecture this result is connected with our findings about the difficulties multimodal encoders have in grounding negation @xcite ; we believe further work is needed to understand the role of negation in visual dialogues. finally, we highlight the importance of going beyond the simple chair metric to evaluate the impact of hallucination. by running quantitative and qualitative analysis on human dialogues from the guesswhat?! test set, we found that vlp is the model that generates less severe and more human-like hallucinations. further work is needed to design new decoding strategies for natural language generation systems and to explore the relation between hallucination and repetitions, another major issue that heavily affects the quality of machine-generated data as recently highlighted in @xcite . moreover, as the example in figure 1 (right) shows, attribute hallucination plays an important role in the quality of the generated output, and it has not received much attention from the research community.