M ed- HALT : Medical Domain Hallucination Test for Large Language Models (2023)

Abstract: This research paper focuses on the challenges posed by hallucinations in large language models (LLMs), particularly in the context of the medical domain. Hallucination, wherein these models generate plausible yet unverified or incorrect information, can have serious consequences in healthcare applications. We propose a new benchmark and dataset, Med-HALT (Medical Domain Hallucination Test), designed specifically to evaluate and reduce hallucinations. Med-HALT provides a diverse multinational dataset derived from medical examinations across various countries and includes multiple innovative testing modalities. Med-HALT includes two categories of tests reasoning and memory-based hallucination tests, designed to assess LLMs’ problem-solving and information retrieval abilities. Our study evaluated leading LLMs, including Text Davinci, GPT-3.5, LlaMa-2, MPT, and Falcon, revealing significant differences in their performance. The paper provides detailed insights into the dataset, promoting transparency and reproducibility. Through this work, we aim to contribute to the development of safer and more reliable language models in healthcare. Our benchmark can be found at medhalt.github.io

## introduction advancements in artificial intelligence, particularly in the area of large language models (llms) @xcite , have led to transformative applications across various domains, including healthcare @xcite . these models possess the ability to understand and generate human-like text, by learning patterns from vast corpora of text data. and making them valuable resources for medical professionals, researchers, and students. @xcite @xcite despite their impressive capabilities, they are also subject to unique challenges such as hallucination. @xcite , where they generate plausible & confident yet incorrect or unverified information. such hallucinations may be of minimal consequence in casual conversation or other contexts but can pose significant risks when applied to the healthcare sector, where accuracy and reliability are of paramount importance. misinformation in the medical domain can lead to severe health consequences on patient care and outcomes, the accuracy and reliability of information provided by language models can be a matter of life or death. they pose real-life risks, as they could potentially affect healthcare decisions, diagnosis, and treatment plans. hence, the development of methods to evaluate and mitigate such hallucinations is not just of academic interest but of practical importance. efforts have been taken to mitigate the occurrence of hallucinations in large language models @xcite @xcite , but not in the medical field. the purpose of this research work is to address the issue of hallucination in large language models specifically within the medical domain. we propose a novel dataset and benchmark, named med-halt (medical domain hallucination test), a comprehensive evaluation framework designed to measure, and evaluate hallucination in these models. more specifically, it enables researchers to assess the performance of new models, identify and mitigate potential hallucination risks, and ultimately enhance the safety and reliability of these models in critical medical applications.to the best of our knowledge, this dataset and benchmark is the first of its kind to evaluate the hallucinations of llms in the medical domain. the framework is divided into two categories of hallucination tests, namely the reasoning hallucination tests and the memory-based hallucination tests. the former category is designed to assess how well an llm can reason about a given problem by means of false confidence test (fct), none of the above (nota) test, and fake questions test (fqt). the memory-based hallucination tests, on the other hand, focus on evaluating the model's ability to retrieve accurate information from its en-coded training data, a critical task in the medical domain where information needs to be accurate, reliable, and easily retrievable. throughout this research paper, we evaluate and compare the performance of various large language models, including text davinci @xcite , gpt-3.5, llama-2 (touvron et al., 2023) , mpt (mosaicml, 2023), falcon @xcite . by presenting the results and analysing their strengths and weaknesses, we aim to provide an in-depth analysis of their hallucination tendencies within the medical domain. we hope to contribute to the development of more reliable and trustworthy language models in the medical field. fig. 1 shows the overview of the framework. ## datasets statistics med-halt consists of seven datasets. in total, there are 18,866 samples per rht task, with each sample having an average of 238.0 words. moreover, there is also a separate pubmed portion which includes 4,916 samples per mht task, with an average of 37.0 words per sample. the primary details for each of these datasets, along with the corresponding tasks in med-halt, are presented in pubmed : the pubmed dataset, a part of the med-halt framework, includes 4,916 samples derived from the comprehensive archive of life sciences and biomedical information, pubmed. this dataset significantly enhances the diversity of med-halt, providing a rich resource for extracting medically relevant, scholarly content and insights. ## types of hallucination evaluated the med-halt framework proposes a two-tiered approach to evaluate the presence and impact of hallucinations in generated outputs. ## exploratory analysis for the exploratory analysis, we randomly sampled 30% of questions from each exam dataset and pubmed articles. to ensure diversity and balance, we stratified our sampling by country, type of exam, and difficulty level of the questions. ## conclusion this research advances our understanding of hallucination in large language models (llms) within the medical domain, introducing the med-halt dataset and benchmark as a comprehensive tool for evaluating and mitigating such issues. our comparative analysis of models, including openai's text-davinci, gpt-3.5, llama-2, and falcon, has revealed considerable room for improvement.