On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? (2022)

Abstract: Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination. In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models? We conduct a comprehensive human study on both existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Our study reveals that the standard benchmarks consist of > 60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them. We make our annotations publicly available for future research.

## introduction knowledge-grounded conversational models, powered by large pre-trained language models @xcite @xcite , are well-known to generate factually incorrect statements, a phenomenon commonly called hallucination @xcite . a large commonality in the majority of prior work seeks to address hallucination by ameliorating the model @xcite @xcite ), but no attempt has been made so far to audit the conversational benchmarks to the best of our knowledge. on one hand, knowledge-grounded conversational benchmarks may contain hallucinations due to error-prone collection protocols, or due to a design framework that encourages informativeness over faithfulness. existing dialogue systems are typically trained on corpora crowd-sourced through online platforms @xcite @xcite . with loose 1 https://github.com/mcgill-nlp/faithdial figure 1 : an example of a hallucinated conversation from the wizard of wikipedia dataset @xcite . the wizard (yellow) is hallucinating information that cannot be inferred from the knowledge-snippet: hallucinated subjective content (red) and hallucinated objective content (blue). in this work, drawing insights from the linguistic coding system for discourse phenomena @xcite and evaluation frameworks such as begin @xcite and ais @xcite , we annotate responses from the three widely-used knowledge-grounded conversational benchmarks: wizard of wikipedia @xcite , cmu-dog @xcite and topi-calchat @xcite . our analysis reveals surprisingly that more than 60% of the responses are hallucinated in the three datasets, with major hallucination modes that manifest principally through the expression of subjective information (e.g., thoughts, beliefs, feelings, intentions, personal experiences) and the expression of unsupported objective factual information. further, to understand if neural conversational models make this hallucination more severe, we annotate responses generated by several state-of-the-art models, including ones that are designed to alleviate hallucinations. we find that the generated responses consist of an even larger portion of hallucinations, in comparison with the training data. our findings question the quality of current conversational datasets, their appropriateness to train knowledge-grounded conversational systems, and the robustness of existing models. ## hallucinations in benchmarks we conduct a human study on three english crowdsourced knowledge-grounded conversational benchmarks: wizard of wikipedia (wow), cmu-dog and topicalchat. these datasets consist of dialogues between two speakers, where the goal is to communicate information about particular topics while speakers are presented with a knowledge snippet relevant to the current turn. more details about these datasets are provided in §a. response classification taxonomy following the definitions of the begin taxonomy @xcite and the ais framework @xcite of evaluating response attribution, we annotate each response based on whether it can be inferred exclusively from the knowledge-snippet as follows: entailment: a response is fully supported by the knowledge, i.e., any information it contains must be attributed to the knowledge. hallucination: a response's factual correctness cannot be fully verified from the knowledge-snippet (even if it is true in the real world). more specifically, personal opinions, experiences, feelings, internal assessments of reality that cannot be attributed to the information present in the source document, are considered hallucinations. partial hallucination: part of the response is hallucinated while the rest is entailed by the source knowledge. generic: a response that is vague and does not convey any factual information such as "sounds good" or "i'm not sure about that". uncooperative: an entailed response that does not follow the principles of conversational cooperation according to gricean maxims @xcite . the response may be purposefully misleading, or showing a general unwillingness to cooperate with the interlocutor, resulting in an incoherent communication. to understand the linguistic nature of hallucinations, we further annotate responses based on a linguistic coding system for discourse phenomena, dubbed verbal response modes (vrm; stiles 1992). concretely, we label a turn with the following speech acts: disclosure, edification, advisement, confirmation, question and acknowledgement (ack.). ## hallucination amplification in models next, we investigate how much models amplify the hallucination phenomenon at inference time. we consider a range of representative models: ## related work hallucination in neural language generation has recently attracted the attention of several researchers in many areas including neural machine translation (nmt) @xcite and summarization @xcite . hallucinations in knowledge-grounded neural dialogue generation is instead a nascent research problem @xcite @xcite . most existing works focus on avoiding hallucinations in generated outputs by introducing more robust training approaches. dziri et al. (2021a) who introduce frameworks for quantifying attribution in dialogue systems, whereas we conduct a much finer-grained manual analysis on multiple benchmarks and models. ## conclusion our investigations demonstrate empirically that hallucination is a prevalent issue in both dialog benchmarks and models. our analysis on three widely used benchmarks reveals that they are rife with hallucinations, and the most common strategies people use are disclosure and edification. moreover, we show that conversational models trained on these benchmarks not only hallucinate but also amplify hallucinations, even the models that were designed to alleviate this issue. this calls for a clean high-quality data release and careful design of trustworthy conversational systems. before then, we strongly advocate practitioners to look at samples of any dataset-in order to uncover actionable insights-prior to their use or public release.