On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Abstract
Knowledge-grounded conversational models are known to suffer from producing factually invalid statements, a phenomenon commonly called hallucination.
In this work, we investigate the underlying causes of this phenomenon: is hallucination due to the training data, or to the models?
We conduct a comprehensive human study on both existing knowledgegrounded conversational benchmarks and several state-of-the-art models.
Our study reveals that the standard benchmarks consist of > 60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations.
Our findings raise important questions on the quality of existing datasets and models trained using them.
We make our annotations publicly available for future research.
Introduction
Knowledge-grounded conversational models, powered by large pre-trained language models (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020), are well-known to generate factually incorrect statements, a phenomenon commonly called hallucination (Dziri et al., 2021b; Rashkin et al., 2021b).
A large commonality in the majority of prior work seeks to address hallucination by ameliorating the model (Shuster et al., 2021;
Mielke et al., 2020;
Dziri et al., 2021a; Rashkin et al., 2021b), but no attempt has been made so far to audit the conversational benchmarks to the best of our knowledge.
On one hand, knowledge-grounded conversational benchmarks may contain hallucinations due to error-prone collection protocols, or due to a design framework that encourages informativeness over faithfulness.
Existing dialogue systems are typically trained on corpora crowd-sourced through online platforms (Dinan et al., 2018; Gopalakrishnan et al., 2019; Moon et al., 2019).
With loose incentive to come up with faithfully-grounded utterances on the provided knowledge, crowdworkers may ignore knowledge-snippets altogether, use their personal knowledge or sometimes assume a fictional persona, resulting in conversations that are rife with subjective content and unverified factual knowledge.
Figure 1 shows a hallucinated conversation from the W O W dataset (Dinan et al., 2018),
On the other hand, neural conversational models are not necessarily designed to generate faithful outputs, but to mimic the distributional properties of the data.
This kind of optimization will likely push the models to replicate and even amplify the hallucination behaviour at test time (Bender et al., 2021).
The presence of even few hallucinated responses may skew the data distribution in a way that curbs the model’s ability to generate faithful responses (Kang and Hashimoto, 2020).
In this work, drawing insights from the linguistic coding system for discourse phenomena (Stiles, 1992) and evaluation frameworks such as
BEGIN (Dziri et al., 2021b) and AIS (Rashkin et al., 2021a), we annotate responses from the three widely-used knowledge-grounded conversational benchmarks: Wizard of Wikipedia (Dinan et al., 2018), CMU-D O G (Zhou et al., 2018) and T OPICAL C HAT (Gopalakrishnan et al., 2019).
Our analysis reveals surprisingly that more than 60% of the responses are hallucinated in the three datasets, with major hallucination modes that manifest principally through the expression of subjective information (e.g., thoughts, beliefs, feelings, intentions, personal experiences) and the expression of unsupported objective factual information.
Further, to understand if neural conversational models make this hallucination more severe, we annotate responses generated by several state-of-the-art models, including ones that are designed to alleviate hallucinations.
We find that the generated responses consist of an even larger portion of hallucinations, in comparison with the training data.
Our findings question the quality of current conversational datasets, their appropriateness to train knowledge-grounded conversational systems, and the robustness of existing models.
Hallucinations in Benchmarks
We conduct a human study on three English crowdsourced knowledge-grounded conversational benchmarks: Wizard of Wikipedia (W O W), CMUD O G and T OPICAL C HAT .
These datasets consist of dialogues between two speakers, where the goal is to communicate information about particular topics while speakers are presented with a knowledge snippet relevant to the current turn.
More details about these datasets are provided in §A.
Response Classification Taxonomy
Following the definitions of the BEGIN taxonomy (Dziri et al., 2021b) and the AIS framework (Rashkin et al., 2021a) of evaluating response attribution, we annotate each response based on whether it can be inferred exclusively from the knowledge-snippet as follows: Entailment : a response is fully supported by the knowledge, i.e., any information it contains must be attributed to the knowledge.
Hallucination : a response’s factual correctness cannot be fully verified from the knowledge-snippet (even if it is true in the real world).
More specifically, personal opinions, experiences, feelings, internal assessments of reality that cannot be attributed to the information present in the source document, are considered hallucinations.
Partial Hallucination : part of the response is hallucinated while the rest is entailed by the source knowledge.
Generic : a response that is vague and does not convey any factual information such as “ Sounds good " or “ I’m not sure about that ".
Uncooperative : an entailed response that does not follow the principles of conversational cooperation according to Gricean maxims (Grice, 1989).
The response may be purposefully misleading, or showing a general unwillingness to cooperate with the interlocutor, resulting in an incoherent communication.
To understand the linguistic nature of hallucinations, we further annotate responses based on a linguistic coding system for discourse phenomena, dubbed Verbal Response Modes (VRM; Stiles 1992).
Concretely, we label a turn with the following speech acts: Disclosure , Edification , Advisement , Confirmation , Question and Acknowledgement (Ack.) .
Table 1 displays the definition for each VRM type.
We opted for the VRM taxonomy as it offers a simple way of codifying responses into categories that are sufficient for our analysis whereas one can also opt for a more demanding annotation scheme (Bunt et al., 2020).
2. Human Evaluation Study
We follow a two-stage annotation protocol where we first ask two linguists to judge the attribution of 200 randomly sampled train responses with respect to the source knowledge.
Details about experts can be found in §D. For inter-annotator agreement, we measure Fleiss’ Kappa scores on both BEGIN and VRM.
W O W achieved 0.89 on BEGIN and 0.78 on VRM, indicating substantial agreement.
Annotations on CMU-D O G and T OPICAL C HAT achieved nearly similar agreement (See §E).
The high agreement scores align with the findings in AIS on W O W (Rashkin et al., 2021a).
The second round corresponds to a large-scale annotation of 4K randomly sampled train responses using non-expert annotators from AMT.
This round is crucial to ensure that the obtained results from the experts are reliable enough to draw conclusions about the quality of the data.
As human annotation is expensive, we perform the nonexpert annotations only on the W O W benchmark while restricting ourselves to expert annotations on CMU-D O G and T OPICAL C HAT data.
We choose W O W over the other two datasets as the source knowledge is more amenable to faster annotation (T OPICAL C HAT : 300 words > CMU-D O G:words > W O W: 27 words).
Details about our AMT task design and how we ensure data quality can be [Figure] found in §F.
In total, we selected 4 trusted workers to annotate the 4k responses.
To compute the inter-annotator agreement, we assign three workers per response in a secondary task, and ask each of them to judge 500 responses.
Reported Fleiss’
Kappa agreements were 0.75 for BEGIN and 0. for VRM. Although substantial, the agreement is lower than the experts’ one and this is expected as they have stronger linguistic background.
We seek to answer the following questions:
(Q1)
How much hallucination exists in the benchmarks?
Figure 2 shows the breakdown of each BEGIN categoty in W O W and compares expert annotations versus AMT workers.
Surprisingly, W O W is fraught with hallucinations.
Expert annotations on 200 responses show that hallucinated responses are largely mixed with faithful content ( 42 .
3% v.s. 19 .
7% fully hallucinated responses), which amounts to 62% hallucinations in total.
These results generalize even on larger data; we can see that the portion of hallucinated responses increased to 74 .
4% when evaluated on 4K samples.
Our analysis shows similar trends on the CMU-D O G and T OPICAL C HAT benchmarks (Figure 3).
CMU-D O G contains 61 . 4% responses that are purely hallucinated against only 16 . 2% responses that are fully entailing the source knowledge and T OPICAL C HAT has similar results ( 63 . 9% hallucination v.s. 22 .
9% entailment). Exemplars of hallucinated responses are depicted in §J. These findings raise the question on the qual- [Table] ity of dialogue datasets.
(Q2)
What are the hallucination strategies used in human-human data?
Figure 2 and Figureshow the VRM breakdown for each BEGIN category in the three benchmarks.
We make the following observations: The majority of hallucinations belong to disclosure (i.e., subjective information) in all benchmarks (50.9%, 56.2% and 61.5% in
W O W, CMU-D O G and T OPICAL C HAT respectively).
Although the strategy of sharing subjective information such as thoughts, opinions and feelings is natural in conversations, it often comes at a cost of ignoring the knowledge snippet in these datasets.
Moreover, edification is also a common phenomenon in hallucinated responses, suggesting that humans not only discuss subjective informa- [Figure] tion but also bring extra unsupported facts, either true or false.
Other linguistic modes are also associated with hallucinations such as acknowledging unsupported claims or asking irrelevant questions.
Conversely, entailment responses have high percentage of edification ( > 70% ) with information inferred from the knowledge snippet.
Hallucination Amplification in Models
Next, we investigate how much models amplify the hallucination phenomenon at inference time.
We consider a range of representative models:
• GPT2 (Radford et al., 2019; Wolf et al., 2019) is an autoregressive model which takes as input a concatenation of the knowledge and the history.
• DoHA (Prabhumoye et al., 2021) builds a BARTbased conversational model (Lewis et al., 2020) for knowledge-grounding, with a two-view attention mechanism to handle separately the encoded document and the history during generation.
• CTRL (Rashkin et al., 2021b) augments the GPT2 model with control tokens (Keskar et al., 2019) that guide the generation towards less subjective and more entailed content.
We fine-tune each model on the benchmarks and use nucleus sampling (Holtzman et al., 2019) with p = 0 .
6 for decoding (more implementation details are in §B).
As seen in Table 2, CTRL is the best model followed by DoHA based on the hallucination ratio.
Table 6 in §L shows a sample of generated responses.
Similar to the analysis in §2, we task the same two linguists to analyze modelgenerated responses for 200 randomly-selected test [Figure] [Table] samples from each benchmark.
(Q3)
Do state-of-the-art conversational models amplify hallucination?
Table 2 shows the degree of amplification across different models trained on the three benchmarks.
Numbers report the percentage of each class in the data.
Contrasting this with human gold responses, the models not only hallucinate but also amplify the percentage of hallucinations, except CTRL on W O W.
For example, GPT2 amplifies full hallucination by 19 .
2% in W O W, 15% in CMU-D O G and 15 .
1% in T OPICAL C HAT .
Conversely, it reduces entailment by 17 .
3% and 11 .
9% respectively.
This suggests that hallucination patterns are easier to learn than entailment.
Among the three, CTRL hallucinates the least at the expense of producing a high number of uncooperative responses. Although these responses are entailing the knowledge, they are not coherent with the history. A closer inspection shows that most uncooperative responses are extractive, i.e., they copy big chunks of the evidence without adapting the content to the history or they just output an exact copy of the entire evidence. This is also reflected in high ROUGE scores between the response and the knowledge, corroborating the extractive nature of CTRL compared to the gold responses.
This behavior is not surprising as CTRL was optimized to maximize the overlap with the knowledge.
Overall, these results demonstrate that hallucination is not only a reflection of training data issues, but also a consequence of the weaknesses of models.
We hypothesize that there are multiple factors that can contribute to the models’ deficiencies:
First, the exposure bias (Ranzato et al., 2016) caused by teacher forcing can make hallucination worse as the model may over-rely on previously predicted words which in turn can aggravate error propagation. Second, maximum likelihood estimation can be fragile to noisy data points as it necessitates models to assign high probability mass to all test references, resulting in unstable behavior—a fact observed in machine summarization (Kang and Hashimoto, 2020). Moreover, we link this issue to the decoding strategies used at test time. We conjecture that models—when conditioned on factual knowledge—often assign the highest probability mass to the correct response and sampling based on other distributions (e.g. top-k or nucleus) may invite hallucination in the generation process.
And lastly, we hypothesise that the behavior of these models is ultimately shaped by the bias learned from internet text during pre-training (Nadeem et al., 2021).
We leave investigating the role of each factors to hallucination amplification for future work.
(Q4) What are the hallucination strategies used by models?
Surprisingly, different models use different strategies for hallucination.
While DoHA and GPT2 predominantly rely on and amplify disclosure , CTRL relies on edification .
This is because CTRL is trained explicitly to avoid pronouns (a crucial ingredient for disclosure) and to generate entailed responses.
As a side-effect, it ends up amplifying uncooperative responses (by 33 . 5% , 12 . 9% and 20 . 2% in W O W and CMU-D O G as seen in Table 2).
Full results of all models and datasets are in Figure 6, 7 and 8 in §K.
Related Work
Hallucination in neural language generation has recently attracted the attention of several researchers in many areas including neural machine translation (NMT) (Raunak et al., 2021; Wang and Sennrich, 2020) and summarization (Durmus et al., 2020; Kang and Hashimoto, 2020).
Hallucinations in knowledge-grounded neural dialogue generation is instead a nascent research problem (Mielke et al., 2020; Shuster et al., 2021; Dziri et al., 2021a; Rashkin et al., 2021b). Most existing works focus on avoiding hallucinations in generated outputs by introducing more robust training approaches.
Dziri et al. (2021a) propose a model that uses facts supplied by a knowledge graph to reduce entity-based hallucinations in generated responses. Rashkin et al. (2021b) add control tokens at training time to control generation towards more objective sentences and faithful sentences.
Closest to our work are Dziri et al.
(2021b)
and Rashkin et al.
(2021a) who introduce frameworks for quantifying attribution in dialogue systems, whereas we conduct a much finer-grained manual analysis on multiple benchmarks and models.
Conclusion
Our investigations demonstrate empirically that hallucination is a prevalent issue in both dialog benchmarks and models.
Our analysis on three widely used benchmarks reveals that they are rife with hallucinations, and the most common strategies people use are disclosure and edification .
Moreover, we show that conversational models trained on these benchmarks not only hallucinate but also amplify hallucinations, even the models that were designed to alleviate this issue.
This calls for a clean high-quality data release and careful design of trustworthy conversational systems.
Before then, we strongly advocate practitioners to look at samples of any dataset—in order to uncover actionable insights—prior to their use or public release.
Acknowledgements
We are grateful to the anonymous reviewers for helpful comments.
This research is supported by the Mila-IBM grant and the Alberta Machine Intelligence Institute Fellow Program.
We also acknowledge the support of the NSERC Discovery grant and the Facebook CIFAR AI Chair program.
Impact Statement & Ethics Annotation Risks
The benchmarks we audit were collected through AMT and thus may contain some disturbing examples including racist or even expletive phrases.
Annotators were also asked to judge the outputs of several state-of-the-art conversational systems which may be in turn toxic and insensitive.
We acknowledge the psychological distress that this may present to workers (Arditte et al., 2016).
Therefore, we alert workers by adding the following warning in italic text in each HIT: If this
HIT causes you emotional distress or elicit feelings of trauma, please feel free to skip it.
Deployment Risks
Our analytical study reveals that a large portion of standard knowledgegrounded dialogue benchmarks is hallucinated, leading us to reflect on the potential harm of lowquality data releases for conversational models.
In recent years, the conversational AI market has seen a proliferation of a variety of applications—which are powered by large pre-trained LMs—that span across a broad range of domains, such as customer support, education, e-commerce, health, entertainment, etc (Vakulenko et al., 2021).
Ensuring that these systems are trustworthy is key to deploy systems safely at a large scale in real-world application, especially in high-stake domains (Sambasivan et al., 2021).
However, even if we come up with a model that is robust enough against hallucination, it will be ultimately bounded by the data quality.
We argue that fixing the models or the data to enforce faithfulness is a highly non-trivial task without an in-depth understanding of the various sources of hallucination.
Our work thus represents the first effort to gain such an understanding and to inform the community about the unreliability of the existing benchmarks and models.
As result, we believe it is important to raise these insights to the broader community.
A Datasets
We conduct our analysis on the following datasets:
Wizard of Wikipedia: composed of dialogues between a “wizard” and an “apprentice”, where the goal of the wizard is to communicate information about a particular topic and the apprentice is tasked to seek information about that topic.
At each turn, the wizard is presented with a knowledge snippet from Wikipedia and asked to form an utterance.
We filter data points in which the wizard did not explicitly select a passage as knowledge for the response.
In total, the dataset is comprised ofgrounded-responses in train, 8800 valid andtest.
CMU-DoG: All conversations focus only on the movie domain.
Each response is grounded on a section from Wikipedia.
Workers are asked to either persuade the other speakers to watch the movie using information from the Wikipedia section or to discuss the content of the document with them.
In total, there are 78136 grounded responses in train, 13800 in valid and 13796 in test.
TopicalChat: Contrary to CMU-D O G, T OPICAL C HAT conversations are about a variety of topics.
Workers are provided relevant facts from Reddit, Wikipedia and news articles.
The collection process corresponds to two scenarios: symmetric and asymmetric.
In the symmetric scenario, workers have access to the same source knowledge and in the asymmetric scenario, they have access to different sources.
In total, the dataset hasgrounded responses in train, 23601 in valid and 23623 in test.
B Implementation Details
GPT2:
This model was implemented using the Pytorch Huggingface Transformers library (Wolf et al., 2020) and the Pytorch-lightning library 2 .
To train the models, we use the Adam optimizer (Kingma and Ba, 2015) with Dropout (Srivastava et al., 2014) on a batch size of 32 with a learning rate of 6 .
25 × 10 − 5 that is linearly decayed.
The maximum dialogue history length is set to 3 utterances.
The model early-stops at epoch {7, 8, 8} respectively for W O W, CMU-D O G and T OPICAL C HAT .
The average runtime is {1.5, 3, 3} hours for W O W, CMU-D O G and T OPICAL C HAT respectively.
DoHA:
We use the pre-trained model on CMUDoG that is publicly available 3 .
However, since no models trained on W O W and T OPICAL C HAT have been released, we follow closely the training procedure described in Prabhumoye et al. and we train two models.
The average runtime of these models is {5, 10} hours for W O W and T OPICAL C HAT respectively.
CTRL:
We implement the model ourselves since the code and the model were not released by the authors. We follow training details in Rashkin et al.
(2021b) and implement this model using the Pytorch Huggingface Transformers library and
the
Pytorch-lightning library.
Additionally, we had multiple discussions with the authors to make sure that our implementation is accurate.
We save the best model based on the validation set, for all datasets.
Training for all models is done on an Nvidia V100 GPU 32GB and for inference, we use nucleus sampling with p=0.6.
C Definition of VRM
Table 3 contains VRM definitions with examples.
D Expert Annotation
The two experts were students with linguistics background, fluent in English, and were trained for the task by exchanging rigorous discussions with the authors.
As part of this stage, they were required to write justifications for 50 samples articulating the reasoning for the provided ratings.
The collected justifications were helpful in understanding the reasoning used to reach their ratings and in laying the groundwork for designing the second round of annotations.
E Inter-annotator Agreement on Gold Responses
Table 4 contains the Fleiss kappa scores for CMUD O G and T OPICAL C HAT .
F AMT Human Annotation Task Design
To streamline the process for raters we break down the task into hierarchical (yes/no) questions.
We summarize this procedure below, and provide the exact questions in §G. First, we ask [Table] [Table] annotators to judge whether the response contain information that is not supported by the source.
If yes, we ask them to indicate the type of the unsupported information (e.g., unsupported opinion, unsupported fact, etc).
In a followup question, we ask them to indicate whether there are any supported information besides the hallucinated content.
If the response was not hallucinated, we present them with two follow-up questions about whether the response is entailing the source or generic.
Finally, if the response entails the source, we ask whether it is coherent with the history.
AMT Data Quality
To access the initial staging round in AMT, workers have to pass a qualification test by answering correctly 14 questions about
BEGIN and VRM.
Moreover, they had to be situated in the United States and Canada. Before being granted access to the main annotation task, workers would have access only to a small pilot round (batch size ∼ 50 HITs).
In this round, we carefully inspect each of the workers annotations for adherence to the instructions, and provide feedback via email to those who committed errors.
At the end of this round, we revoke access for workers who provide poor quality annotations.
Next, we launch the main annotation stage which is larger (batch size ∼ [-
 HITs) and more efﬁcient.-]400 HITs).
We perform daily manual inspection and we send detailed feedback to workers who commit persistent error patterns.
We reject poor quality work in this stage and repeated rejections lead to blocking the workers from the task indefinitely.
In total, we ended up withworkers annotating the 4k responses.
The workers were informed that their annotations would be used for research purposes and their workers ID would be anonymous when we release the data.
G AMT Human Instructions
AMT Human annotation interfaces are depicted in
Figure 4 and Figure 5.
We pay workers an hourly wage around 18-20 USD which is above the minimum wage rate.
Workers were asked the following questions:
Does the Wizard’s response contain other information that is NOT supported by the evidence?
(E.g., facts, opinions, feelings)?
(a) If the response is hallucinated, what is the type of the unsupported information?
(expressing a personal experience, expressing an opinion, expressing feelings, expressing unsupported facts, giving advice, acknowledging with information from the human) (b) Besides unsupported information, does the Wizard’s response contain thoughts/opinions/feelings/facts that are supported by the Evidence? 2.
If the response is not hallucinated, is it faithful to the source or generic?
(Faithful, Generic) 3.
If the response if faithful, is it cooperative with the Human’s response?
H Limitation
The main goal of this work is to present a data quality audit by gaining an in-depth understanding of the various types of hallucination in both gold and machine-generated responses.
We do not investigate the root causes of hallucination in the models.
Also, we limit our analysis to only English
Benchmarks.
Future studies can extend our work to explore the main causes of hallucination in the models and study the problem of hallucination in multilingual datasets.
I Hallucination in CMU-DoG and TopicalChat
Figure 3 shows the hallucination breakdown in
CMU-D O G and T OPICAL C HAT benchamrks.
J Hallucinated Human-Human Responses
Table 7 contains hallucinated gold responses from
W O W, CMU-D O G and T OPICAL C HAT .
K Breakdown of BEGIN and VRM in Machine-generated Responses
Figure 6, 7 and 8 display the distribution of BEGIN and VRM in GPT2, D O HA and CTRL trained on the three benchmark.
L Machine-generated Responses
Table 6 contains a sample of generated responses from GPT2, D O HA and CTRL on the W O W and
CMU-D O G.
[Figure]
[Figure] [Figure] [Figure] [Table]
[Table] [Table]
search requires more
[Table]
power than it took to send [Table] eleven other
[Table]
the moon [Table] Google
[Table] traffic by tracking how fast [Table] devices are moving on the road instead of hiring people to mow the lawns around their headquarters.
Google uses hundreds of live goats.
On 16th August
[Table] Google went down for about five minutes [Table] and took 40% of web traffic with it.
When there is a disputed border, Google maps tailors its maps to the claims of each country where the Internet browser is located
[Table]
Google provides online related services and products, which includes online ads, search engine and cloud computing
[Table]