TU W ien at S em E val-2024 Task 6: Unifying Model-Agnostic and Model-Aware Techniques for Hallucination Detection (2024)

Abstract: This paper discusses challenges in Natural Language Generation (NLG), specifically addressing neural networks producing output that is fluent but incorrect, leading to “hallucinations”. The SHROOM shared task involves Large Language Models in various tasks, and our methodology employs both model-agnostic and model-aware approaches for hallucination detection. The limited availability of labeled training data is addressed through automatic label generation strategies. Model-agnostic methods include word alignment and fine-tuning a BERT-based pretrained model, while model-aware methods leverage separate classifiers trained on LLMs’ internal data (layer activations and attention values). Ensemble methods combine outputs through various techniques such as regression metamodels, voting, and probability fusion. Our best performing systems achieved an accuracy of 80.6% on the model-aware track and 81.7% on the model-agnostic track, ranking 3rd and 8th among all systems, respectively.

## introduction in natural language generation (nlg), the tradeoff of prioritising fluency over accuracy results in neural networks generating "hallucinations" -outputs fluent but factually inaccurate. the automatic identification of such errors represents a substantial challenge @xcite . the shroom shared task on hallucination detection @xcite highlights concerns about the practical utility of fluently generated yet inconsistent outputs. in the shroom shared task, large language models' (llms) outputs for definition modeling (dm), machine translation (mt), and paraphrase generation (pg) tasks are presented with input source text and corresponding 'gold' reference text. notably, for pg, the input source text serves as the reference 'gold' text. while the training dataset lacks labels, an issue which is addressed in section 3, the validation dataset includes binary labels of hallucination or not hallucination and hallucination probability of 0 to 1, corresponding to hallucination and not hallucination, respectively, for the llm's output. these assessments, based on five annotators' evaluations, rely on determining if the model's output is supported by the 'gold' reference, from either the 'gold' target text, source text, or both, depending on the task (dm, mt, or pg). the shroom dataset is categorised into two tracks: model-agnostic and model-aware. the model-aware track, in contrast to the modelagnostic, includes the specific llm responsible for the provided output. this paper introduces methods tailored to both tracks and since the modelagnostic techniques can be applied to both, evaluations for these methods are conducted on both test datasets to provide a comprehensive analysis. the model-agnostic methods entail employing word alignment to establish semantic similarity between the model output and the 'gold' reference as well as fine-tuning a bert-based model for hallucination detection. on the other hand, model-aware approaches delve into the analysis of hidden states and attention flow within the model architecture. ultimately, a diverse set of ensemble techniques, comprising logistic regression with binary labels, linear regression with raw probabilities, voting, and probability fusion, are introduced to amalgamate the proposed methods. ## related work since the tendency of llms to produce incorrect output poses a serious challenge in their application, the task of hallucination detection has recently attracted a variety of research work. model-agnostic approaches include the training of dedicated machine learn-ing models, such as a token-level classifier for detecting hallucination in machine translation @xcite or the bert-based vectara hallucination_evaluation_model foot_1 , the latter which we also use in our model-agnostic experiments (see section 3.2). recent datasets for training and evaluating such models include taskagnostic corpora such as @xcite and hades @xcite as well as datasets focusing on specific generation tasks such as text summarisation @xcite , fact verification @xcite , question answering @xcite , or paraphrase generation @xcite . another set of approaches involves comparing a model's output to some reference using any of a variety of unsupervised similarity metrics, including standard ngram-based measures such as bleu @xcite and rouge @xcite but also distributional similarity metrics such as bertscore @xcite , bartscore @xcite , or discoscore @xcite . a large-scale analysis of the performance of these metrics on hallucination detection have been performed in the recent true survey @xcite . further model-agnostic approaches to hallucination detection include comparing multiple llm responses to a single query @xcite , prompting llms to evaluate the likelihood of their own output being correct @xcite as well as the use of external knowledge bases to assess the faithfulness of model outputs @xcite @xcite model-aware methods for hallucination detection include classifying an llm's hidden layer activations to determine whether the question is answerable @xcite or whether its output is true @xcite . the latter approach is the basis of the saplma system, which we have also used in our experiments for the model-aware track of the shared task (see section 3.3). ## methodology within the scope of hallucination detection, we employed both model-agnostic and model-aware methods. our model-agnostic approaches encompassed rule-based techniques, featuring the applica-tion of string metrics and word embeddings, alongside the fine-tuning of pretrained language models. model-aware methods leveraged the internal data of llms. ## results ## conclusion this paper outlines our systems for the semeval shared task on llm hallucination detection, covering both model-aware and model-agnostic subtasks. our finetuned bert-based model demonstrated strong performance, securing the 3rd rank in the model-aware track and underscoring the efficacy of our approach. notably, our leading system in the model-agnostic track employs a metamodel that integrates predictions from diverse systems, including a finetuned bert-based hallucination detection model, as well as rule-based methodologies and those relying on llm hidden states. the absence of a universally agreed-upon definition for hallucination complicates both human and machine evaluations, as evident in the error analysis of section 6.1. although attempts have been made to systematically define hallucination, such as by @xcite and @xcite , the nlp community's understanding remains broad. this encompasses scenarios where a model outputs entirely false information or information close to the desired output but incomplete. human annotators bring their world knowledge and views, influencing annotations and subsequently affecting model performance. hallucination detection is challenging for both machines and humans, with achieving high inter-annotator agreement proving particularly difficult when the task definition is overly broad. especially tasks like paraphrase generation or definition modeling, where numerous correct outputs are possible, are inherently subjective and tied to the annotator's real-world knowledge and beliefs @xcite . a clearer definition of the annotation task, specifically detailing what constitutes hallucination for that task, could potentially enhance inter-annotator agreement and subsequently improve model performance. detecting hallucinations across various nlp tasks poses a significant challenge. the shroom dataset encompasses three distinct tasks, whereas existing hallucination detection benchmarks often address only a single task.