Do Language Models Know When They’re Hallucinating References? (2024)

Abstract: State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the “model organism” of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as “consistency checks.” Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to “know” when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature.

## introduction despite their unparalleled capabilities, recent large language models (llms) still exhibit a tendency to generate seemingly credible yet incorrect or unfounded information. this phenomenon is often referred to as the "hallucination" problem in the field of natural-language processing (nlp). 1 as one might imagine, the ramifications of these hallucination generations can be profoundly detrimental when these outputs find their way to critical domains such as healthcare, finance, law, or academic publications, where factuality is essential and nonnegotiable. in fact, a recent example underlining the gravity of this issue involved a u.s. judge imposing sanctions on two new york lawyers for submitting a legal brief that included several fictitious case citations that were generated by chatgpt. 2 the are two primary challenges ahead for both researchers and practitioners within the nlp community. the first requires developing a deeper understanding of why these language models resort to fabricating information, while the second demands creating mechanisms that can not only promptly detect but also mitigate, if not completely prevent, inaccurate information in model outputs. to that effect, in this work, we study the problem of hallucinated book and article references related to the field of computer science and present a simple yet effective method to detect hallucinated references without relying on external tools. drawing inspiration from the role of the fruit fly, drosophila melanogaster, as a model organism in biological research, we suggest that the nlp community focus on the study of hallucinated references to better understand and mitigate wider hallucination challenges. these hallucinated references present distinct characteristics that render them suitable for study. first, their automatic classification is more straightforward than other hallucination varieties. 3 as an illustration, our method that leverages a search engine api closely matches of meaning @xcite . additionally, we use the terms hallucinate and fabricate interchangeably throughout the paper. 2 the original newspaper article detailing this incident can be found at this link. @xcite each of four human expert evaluations, in at least 99 out of a sample of 100 references. moreover, the static nature of academic reference titles, combined with their broad online availability (on platforms like google scholar, semantic search, and arxiv), suggests they frequently appear in large, popular language modeling corpora. additionally, many within the research domain already possess the skills and knowledge pertinent to studying these hallucinations. we therefore believe that just as fruit fly studies have enriched our understanding of biology, focusing on these specific reference hallucinations can pave the way for insights and solutions for more complex and challenging hallucination forms. we outline the rest of this work as follows. we are interested in investigating when and why language models produce hallucinated references and what can be done to prevent them. we explore whether llms such as gpt-4 can recognize their own hallucinated outputs without relying on any external tools. while this approach does not fully unravel the reasons behind and solutions to hallucinations, it adds valuable perspective. specifically, if llms can identify their own hallucinations, it implies the root of the issue may not lie in training or representation, but rather in the generation (i.e., decoding) process, given that models inherently possess enough data to potentially lower the rate of hallucinations. our experiments compared different questioning strategies to use the lm to detect its own hallucinations across gpt and llama based lm's. ## preliminaries and background following @xcite , we define "hallucination" as fabricated text that has little or no grounding in the training data. it is worth noting that this is sometimes referred to as open-domain hallucination to distinguish it from closed-domain hallucination (see: @xcite ). 4 our usage of the term hallucination aligns with the open-domain variant. distinguishing groundedness from correctness. the measure of correctness (or factuality) relies upon a comparison with ground-truth answers. previous work on hallucination has blurred the line between groundedness and factuality. (sometimes this distinction is also referred to as honesty versus truthfulness @xcite ). for example, the common misconception that "people use 10% of their brains" might be considered grounded if it is mentioned in the training data and assumed to be a true statement; however, this does not mean that it is factual, as it is not a scientifically correct statement. evaluating groundedness. perfectly evaluating hallucinations would require access to the lm's training data. an advantage of the hallucinated reference problem is ease of (approximate) evaluation in that exact-match web search is a reasonable heuristic for groundedness. this is because the vast majority of article titles present in the training data are included in web search results-articles are meant to be published and shared, and publishers aim to make their work discoverable by search. furthermore, references generally have titles that are specific enough not to spuriously occur on the web. regarding other types of hallucinations, besides article names, which cannot be as easily evaluated, we still hope that our methodology and findings would apply, even if evaluating those types of hallucinations would require access to the training data. ## related work open-domain hallucinations, in the context of gpt-4 discussions @xcite , have garnered attention given their prevalence and associated hazards. @xcite comment: "open domain hallucinations pose more difficult challenges, per requiring more extensive re-search, including searches and information gathering outside of the session." yet, our work provides evidence that addressing these hallucinations can be achieved without turning to external resources. as mentioned, there are multiple definitions of hallucination. in this work, we use the term hallucinations to mean fabricated text that is not grounded in the training data. factually incorrect generations can be decomposed into two types of errors @xcite : grounded errors which may be due to fallacies in the training data (e.g., that people use only 10% of their brains) and ungrounded errors. these two types of errors may need different techniques for remedy. the grounded errors may be reduced by curating a training set with fewer errors or other techniques such as rlhf @xcite . however, the ungrounded errors which we study foot_2 are a fascinating curiosity which still challenge the ai community and one which is not clearly addressable by improving training data. there is comparatively little prior work studying open-domain groundedness like ours. some work (e.g., @xcite in attribution aims to understand which training examples are most influential in a given output. in recent independent work in the health space, athaluri et al. ( 2023 ) did an empirical evaluation of hallucinated references within the medical domain. similar to our approach, they used a google search for exact string match as a heuristic for evaluating hallucinations. our study of hallucinated references enables us to estimate the hallucination rates of different models, and, as discussed in prior work, the hallucination problem interestingly becomes more pressing as models become more accurate because users trust them more @xcite . related recent works include black-box techniques for measuring confidence in lm generations. although these works are targeted at factual confidence, the approaches are highly related to our work. while @xcite use probability estimates drawn from lms, it is straightforward to extend their procedures to generation-only lms like chatgpt using sampling. @xcite show that lms can be used to articulate estimates by generating numbers or words as we do. finally, @xcite perform self-checks in the context of summarizing a document. all of these works use direct queries which influenced the design of our direct queries. due to space limitations, we do not discuss the work studying closed-domain hallucination (e.g., in translation or summarization) but instead refer the reader to recent survey of @xcite . ## methodology: consistency checks we now provide an overview of our simple yet effective consistency check methodology, explaining how we perform a series of direct and indirect queries to detect hallucinated references. foot_3 ## experimental details here, we describe the steps taken to build a corpus of article and book references pertaining to computer science topics for each language model, as well as the automatic labeling heuristic used to annotate these generated references. ## results and discussion in this section, we discuss the performance of the indirect and direct methods using quantitative metrics, and present interesting qualitative findings. ## conclusions open-domain hallucination is an important but slippery concept that is difficult to measure. by studying it in the context of references using search engine results, we can quantitatively compare hallucinations across lms and we can also quantitatively compare different black-box detection methods. of course, for the sole purpose of detection, one could achieve higher accuracy by directly consulting curated publication indexes. however, we hope that our study of black-box self-detection of hallucinated references sheds light on the nature of opendomain hallucination more broadly, where detecting hallucinations is more challenging. it suggests that hallucination is not entirely a problem of training but rather one that can be addressed using only the same internal model representation with different generation procedures. while our direct and indirect query methods are only partially reliable and impractically expensive, we hope they may pave the way towards more efficient methods that generate text with fewer hallucinations and thereby reduce potential harms of language models. there are several directions for future work. 1) improved decoding techniques: an important consequence of our work is the recognition that reducing hallucination may be a problem at generation time. thus, inventing improved (non-black-box) generation procedures is thus a crucial direction for future work. 2) additional indirect questions: one may improve accuracy by adding more indirect questions such as year or venue. these pose additional challenges as a paper with the same title and authors may often appear in multiple venues (e.g., arxiv, a workshop, a conference, and a journal) in different years. 3) generalisability: it would be very interesting to see if the methods we employ could be used to identify other types of opendomain hallucinations beyond references. even though hallucinated references are often given as a blatant example of hallucination, perhaps due to the ease with which they can be debunked, these other types of hallucination are also important. following the investigative interviewing analogy, one way to aim to discover general hallucinations would be to query the lm for "notable, distinguishing details" about the item in question. one could then use an lm to estimate the consistency between multiple answers. however, as mentioned for other domains besides references, it may be impossible to determine whether or not a generation is a hallucination without access to the training set (and unclear even with such access).