H ypo T erm QA : Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLM s (2024)

Abstract: Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. Despite ongoing efforts, hallucinations remain a prevalent challenge in LLMs. The detection of hallucinations itself is also a formidable task, frequently requiring manual labeling or constrained evaluations. This paper introduces an automated scalable framework that combines benchmarking LLMs’ hallucination tendencies with efficient hallucination detection. We leverage LLMs to generate challenging tasks related to hypothetical phenomena, subsequently employing them as agents for efficient hallucination detection. The framework is domain-agnostic, allowing the use of any language model for benchmark creation or evaluation in any domain. We introduce the publicly available HypoTermQA Benchmarking Dataset, on which state-of-the-art models’ performance ranged between 3% and 11%, and evaluator agents demonstrated a 6% error rate in hallucination prediction. The proposed framework provides opportunities to test and improve LLMs. Additionally, it has the potential to generate benchmarking datasets tailored to specific domains, such as law, health, and finance.

## introduction large language models (llms) demonstrate exceptional predictive capabilities for common tokens, but encounter challenges when dealing with rare tokens, especially in mixed contexts @xcite . adversarial effects in real-life scenarios may inadvertently emerge from prompts that combine both common and rare tokens. the question in figure 1 is framed with the valid term "platypus" in the relevant domain and a hypothetical term, "wolf," created using a common what are the similarities and differences between platypus llm and wolf llm? wolf llm is a framework for interactive refinement of llms... platypus llm is a family of finetuned and merged llms... some similarities are... some differences are... figure 1 : hypothetical term sample word within the specified context. the answer can be easily categorized by assessing the llm output: whether it rejects the presence of wolf llm or acknowledges its existence and provides an explanation. indeed, platypus @xcite is an actual language model incorporating a seldomused animal name. in contrast, the wolf language model did not exist when this paper was authored. however, approximately 90% of the time, llms neglect to indicate their lack of information about a hypothetical phenomenon in similar situations (section 4). this characteristic significantly diminishes the reliability of llms and impedes their suitability for deployment in critical decision-making systems. detecting hallucinations is difficult, and it is still an ongoing research problem @xcite . generating examples, like the "wolf llm" example, might help assess the tendency of llms to generate information about nonexistent terms. these examples offer advantages in both revealing and detecting hallucinations: as evident from the provided example, this study specifically targets a particular type of hallucination: the generation of content about non-existent phenomena. however, our approach allows for the creation of more generalized datasets, encompassing factually inaccurate generations @xcite or reliable summarization @xcite . ## benchmark creation figure 2 illustrates the process of our framework, while appendix b contains the terms introduced in this paper. the proposed benchmark process includes two steps to generate the proposed hypothetical terms dataset. the first step includes the generation of an intermediate dataset, which contains hypothetical and valid term couples (see sections 2.1 -2.3). the second step is about transforming these term couples into coherent hypothetical or valid questions (see @xcite ). the gpt-3.5 (openai, 2023b) model was employed for the generation of synthetic data due to its higher performance on common tasks compared to open-source alternatives and its superior cost-efficiency relative to the gpt-4 model (ope-nai, 2023c). the temperature variable was set to zero unless specified otherwise. we generated the dataset in accordance with openai's terms and conditions and usage policies foot_1 . the proposed dataset is designed exclusively for the purpose of preventing and evaluating hallucinations in language models. ## hypotermqa score labeling open-ended long texts is a challenging task. however, by getting insights from the literature about the reasoning @xcite @xcite and reflection @xcite @xcite capabilities of llms, we generate llm agents to automatically decide the label of another llm's response. similar to the factscore @xcite ) framework, we introduce irrelevant labels besides hallucination and valid answer labels. we use the percentage of "valid" labeled answers to "hypothetical questions" as the hypotermqa score. it shows llm's performance to resist hallucination. also, (1 -hypotermqa score) denotes the error rate. the distribution of error rate between hallucination and irrelevant labels or performance on valid questions gives valuable secondary insights about llm performance. however, our focus is on detecting if llms are capable of knowing what they do not know. ## experiments and results for experiments, various combinations of three series of llms are tested or employed as evaluator llm agents: gpt @xcite , llama2 @xcite and orca2 @xcite . gpt and llama2 were chosen due to their high citation rates, while orca2 was selected for its focus on improved training data quality and reasoning performance which may lead to preventing hallucinations. all open-sourced models underwent 4-bit quantization before inference. for 7 and 13 billion parameter-sized models, a single nvidia 16 gb v100 gpu was used, while 3 gpus were employed for 70 billion parameter-sized models. proprietary models were accessed through apis, and models available only through a ui were manually prompted by the authors. a total resource of 2000 gpu hours for open-sourced models and 20$ for api access was used for experiments. llama2-7b, llama2-13b, llama2-70b, and gpt-3.5 models were prompted with all 19,508 questions in the hy-potermqa dataset. for those who need to deal with resource constraints, two smaller subsets were also created. ## conclusion our experiments demonstrated that state-of-the-art models, including gpt-4, exhibit a significant susceptibility to hallucination. increasing the parameter size does not directly mitigate this tendency. notably, the chatgpt model, employing heavy rlhf, outperformed the gpt-4 api, achieving the highest performance. following closely is the orca2:13b model, which emphasizes high-quality pre-training data. our findings suggest that these two training approaches, utilizing heavy rlhf and prioritizing pre-training data quality, are currently the most effective methods for reducing hallucination. however, their respective hypotermqa scores are 11% and 8%. for other models, a common trade-off exists between detecting hallucinated terms and rejecting valid terms, indicating that during the supervised fine-tuning phase, models tend to learn a pattern of refusal rather than truthfulness. our results indicate that current llm training methods are insufficient to prevent hallucinations, emphasizing the need for a fundamental change to ensure the reliability of llms. we anticipate that our framework will facilitate a more targeted focus on the hallucination tendency during language model training, also contributing to the creation of more challenging and specialized benchmarks.