Evaluating Hallucinations in Large Language Models for B ulgarian Language (2023)

Abstract: In this short paper, we introduce the task of evaluating the hallucination of large language models for the Bulgarian language. We first give definitions of what is a hallucination in large language models and what evaluation methods for measuring hallucinations exist. Next, we give an overview of the multilingual evaluation of the latest large language models, focusing on the evaluation of the performance in Bulgarian on tasks, related to hallucination. We then present a method to evaluate the level of hallucination in a given language with no reference data, and provide some initial experiments with this method in Bulgarian. Finally, we provide directions for future research on the topic.

## introduction hallucination in large language models (llms) refers to the generation of non-factual statements or information that cannot be verified from the source. the latest generative language models, such as llama, gpt-4 and other gpt-based models, are known to suffer from hallucination problems. the lack of trustworthiness of the generated outputs of llms is one of the main factors that stop their employment in sectors like education and healthcare, where there are high standards for factual accuracy. while numerous annotated evaluation datasets and benchmarks for evaluating the level of hallucinations exist for the english language, it is not the same for most human languages. evaluation of hallucination in lower-resource languages foot_0 , such as bulgarian, is still an open research problem. due to the lack of annotated data on hallucinations in bulgarian, we chose to work with a zeroresource evaluation method called selfcheckgpt @xcite , which offers an approximate estimation of the amount of hallucinations in the text. we experimented with data from bulgarian matriculation exams, part of the exams dataset @xcite , which we processed to derive prompts for text generation on different school subjects. ## conclusion in this short paper, we demonstrate our work in progress on the task of evaluating the level of hallucination of large language models in bulgarian language. we give definitions of different types of hallucinations and methods for evaluation, make an overview of the related work, and provide some initial experiments. our research is aimed towards more equally spread employment of the latest technology across different languages. researchers working on other lower-resource languages can use our work as a source of ideas generation and inspiration.