AILS - NTUA at S em E val-2024 Task 6: Efficient model tuning for hallucination detection and analysis (2024)

Abstract: In this paper, we present our team’s submissions for SemEval-2024 Task-6 - SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. The participants were asked to perform binary classification to identify cases of fluent overgeneration hallucinations. Our experimentation included fine-tuning a pre-trained model on hallucination detection and a Natural Language Inference (NLI) model. The most successful strategy involved creating an ensemble of these models, resulting in accuracy rates of 77.8% and 79.9% on model-agnostic and model-aware datasets respectively, outperforming the organizers’ baseline and achieving notable results when contrasted with the top-performing results in the competition, which reported accuracies of 84.7% and 81.3% correspondingly.

## introduction in the era that large language models (llms) dominate and shape the trends in the natural language processing (nlp) community, ensuring reliance and accurate functionality of related systems becomes a major concern. hallucinations of language models have recently received lots of attention @xcite @xcite @xcite , questioning the trust that humans can pose in highly intelligent yet probabilistic models. at the same time, recent endeavors formally prove that hallucinations are inherent to llms and thus inevitable in practice @xcite . encompassing the need for detecting and analyzing hallucinations in natural language generation (nlg) tasks, and given the scarcity of related datasets and benchmarks @xcite @xcite , the semeval-2024 task 6 (shroom: a shared-task on hallucinations and related observable overgeneration mistakes) @xcite addresses the presence of semantically unrelated generations with respect to a given input, covering challenging nlp tasks, such as machine translation, definition modelling and paraphrase generation, which are tested both when the underlying model is known or not. to this end, we explore efficient and widely adaptable hallucination detection strategies, tailored to the black-box demands of the problem foot_0 . based on pre-trained models which contain knowledge regarding semantic relationships related to hallucinations, we achieve ∼80% accuracy in hallucination detection by fine-tuning on labeled shroom instances, notably higher than the 74.5% baseline accuracy provided, using an open-source mistral instruction-tuned model foot_1 . specifically, we contribute to the following: our code is available on github foot_2 . the more subtle characteristics of other hallucination types constitute the creation of related benchmarks harder, not to mention techniques for automatic evaluation @xcite @xcite . a limitation tied with such techniques is that in most cases at least model probing is needed, rendering them unusable in cases where the model that produced the reported hallucinations is completely unknown or inaccessible. shroom comes to fill this gap, focusing on semantic faithfulness rather than factuality, while requesting a diverging suite of proposed detection techniques that should even cover cases that the model is not given at all. as a trade-off, implementations on the shroom dataset require the ground-truth output, since the given input does not contain the necessary semantic information to drive decisions on whether a sample is a hallucination or not. our proposed approach only considers given inputs and outputs and does not probe any model, contrary to other black-box techniques @xcite . ## task and dataset description driven by upcoming challenges in the nlg landscape, shroom dataset focuses on the prevalent issues of models generating linguistically fluent but inaccurate (incorrect or unsupported) outputs. participants are tasked with binary classification to identify instances of fluent overgeneration hallucinations in model-aware and model-agnostic tracks. the task encompasses three nlg domains-definition modeling (dm), machine translation (mt), and paraphrase generation (pg)-with provided checkpoints, inputs, references, and outputs for binary classification. the development set includes annotations from multiple annotators, establishing a majority vote gold label. ## methods as the core of our system, we propose a universal and lightweight methodology that leverages wellestablished pre-trained classifiers for hallucination detection. we propose 3 techniques to approach it. ## conclusion in this work, we detect and analyze hallucinations from the shroom dataset introduced in semeval 2024 task 6. we propose a computationally efficient methodology based on fine-tuning models that present semantic cues close to shroom's hallucinations, while model ensembling further boosts results in 3 nlg tasks. our techniques operate in a fully black-box setting, solely requiring inputs and outputs obtained from nlg models. our error analysis demonstrates that our misclassifications are samples of high uncertainty in terms of hallucination probability and, therefore hard to be discerned overall. in total, we aspire that our simple though efficient technique will assist future research in the crucial hallucination detection field. (a) hallucination probability per label (model-agnostic). (b) hallucination probability per label (model aware). closest to the 0.5 threshold the hallucination probability will be (a p('hallucination')=0.4 denotes that 3/5 annotators voted for 'not hallucination', while the rest 2/5 voted for the opposite; on the other hand, a p('hallucination')=0.6 denotes that 3/5 annotators voted for 'hallucination', while the rest 2/5 voted for 'not hallucination'. therefore, the highest uncertainty is observed close to the 0.5 boundary). this requirement is adequately satisfied especially in the model-agnostic case (left plot of figure 7 ), where p('hallucination')=0.6 is the least frequent. further insights can be obtained by looking at figure 8 : when smaller dots are assigned to probabilities close to the 0.5 threshold, the annotators' disagreement is lower, therefore classifying a sample as 'hallucination' or 'not hallucination' is less uncertain. indeed, the less frequently appearing p('hallucination')=0.4 and p('hallucination')=0.6 values in the model-agnostic case denote high separability between hallucinated and non-hallucinated samples. however, highly certain values, such as p('hallucination')=0.0 and p('hallucination')=1.0 only rank in the middle, therefore even if samples are separable with low uncertainty, some minor disagreement persists (1/5 annotators frequently disagrees with the rest). overall, annotators are almost equally confident in classifying 'hallucination' and 'not hallucination' samples, as indicated by the matching pattern regarding label uncertainty for both labels. the model-aware case is more confusing, with p('hallucination')=0.6 scoring the highest; therefore, classifying a sample as 'hallucination' is often accompanied by high uncertainty. on the contrary, uncertainty is lower for the 'not hallucination' label, with p('hallucination')=0.0 ranking as the second most frequent probability. we can conclude that in the model-aware setting of the validation set, annotators are more confident in recognizing the 'not hallucination' class in comparison to the 'hallucination' one. test set as for the test set, figure 9 represents the number of samples per task for both settings. note that the test task distribution is similar to the validation distribution of figure 5with pg being a minority label in all cases. in terms of ground-truth label (hallucination vs not hallucination), figure 10 highlights some label imbalance, rendering the prediction of 'not hallucination' more possible in a random setup for both model-agnostic and model-aware settings. this label distribution matches the validation set label distribution (figure 6 ), for which 'not hallucination' was the majority class as well. hallucination probability per setting is depicted in figure 11 , with lower hallucination values in the range [0, 0.2) being more common. this is again somehow expected since 'not hallucination' is the majority class in test labels. more insights can be obtained by looking at figure 12 , which relates the hallucination probability with the label. especially in the model-agnostic setting (figure 12 -left), the p('hallucination')=0.4 and p('hallucination')=0.6 values are the lowest (smaller dots), while p('hallucination')=0.0 is the highest, denoting that annotators are often certain regarding non-hallucinated samples. certainty for hallucinated samples is somehow lower, as p('hallucination')=1.0 lies somewhere in the middle. nevertheless, p('hallucination')=0.8 is the second more frequent value denoting that 4/5 annotators frequently annotate a sample as 'hallucination'. by observing the right plot of figure 12 , we conclude that certainty is lower in the model-aware setting. even though p('hallucination')=0.0 remains the most frequent probability, indicating high agreement regarding non-hallucinated samples, the p('hallucination')=0.6 value stands in the second place. therefore, many samples classified as 'hallucination' achieved this label with low agreement (3/5 annotators). also, the p('hallucination')=0.2 and p('hallucination')=0.8 are the lowest, denoting that higher agreement (4/5 annotators agreeing) is rare for both 'hallucination' and 'not hallucination' labels. we can assume that model-aware samples are harder by nature to be classified in any of the labels.