D eep P avlov at S em E val-2024 Task 6: Detection of Hallucinations and Overgeneration Mistakes with an Ensemble of Transformer-based Models (2024)

Abstract: The inclination of large language models (LLMs) to produce mistaken assertions, known as hallucinations, can be problematic. These hallucinations could potentially be harmful since sporadic factual inaccuracies within the generated text might be concealed by the overall coherence of the content, making it immensely challenging for users to identify them. The goal of the SHROOM shared-task is to detect grammatically sound outputs that contain incorrect or unsupported semantic information. Although there are a lot of existing hallucination detectors in generated AI content, we found out that pretrained Natural Language Inference (NLI) models yet exhibit success in detecting hallucinations. Moreover their ensemble outperforms more complicated models.

## introduction over the past few years, natural language generation (nlg) models have experienced substantial advancements, particularly due to transformer-based architectures like a generative pretrained transformer (gpt) @xcite . however, two interconnected issues challenge the field: firstly, the tendency of present neural systems to generate incorrect yet smooth outputs and, secondly, the inadequacy of existing metrics in evaluating accuracy over fluency. this causes nlg models to "hallucinate", i.e., produce fluent but incorrect outputs that we currently struggle to detect automatically @xcite . the shared-task on hallucinations and related observable overgeneration mistakes (shroom) has been suggested to address this challenge. in particular, the shroom task aims at addressing the existing gap in assessing the semantic correctness and meaningfulness of nlg models. 1 within the shared task @xcite , one needs to detect grammatically sound english output that contains incorrect semantic information (i.e.,unsupported or inconsistent with the source input) in case there is no labeled training data available. we propose to address the shroom task by leveraging an ensemble of pretrained transformerbased natural language inference (nli) models. the nli models are used to derive features of hallucination probabilities, and then a tree-based gradient boosting model @xcite provides a final decision. our results indicate that nli-based models can be effectively used to detect hallucinations. moreover, the ensemble model highly outperforms the base estimators in correlation with annotators' decisions. to summarize, this work includes the following contributions: additionally, we made the code publicly available. 2 2 background nowadays, it is well known that nlg models often generate coherent outputs that are not faithful to the given input, commonly referred as hallucinations @xcite . hallucination has been studied in a wide range of tasks, including but not limited to summarization @xcite , dialogue generation @xcite and a variety of other nlg tasks. there are several benchmarks for hallucination detection. halueval includes 5,000 general user queries with chatgpt responses and 30,000 taskspecific examples from three tasks, i.e., question answering, knowledge-grounded dialogue, and text summarization @xcite . faithdial is a benchmark for hallucination-free dialogues by modifying hallucinated responses in the wizard of wikipedia (wow) benchmarks @xcite . the shroom shared task organizers went one step further. the shared task was conducted with a newly constructed dataset of 4,000 model outputs labeled by five annotators each, including three nlp tasks: machine translation (mt), paraphrase generation (pg), and definition modeling (dm). participants were asked to detect hallucinations in two different settings: a model-aware track where the organizers also provided a checkpoint to a model that generated the output and a modelagnostic track where they did not. the checkpoints are publicly available on huggingface. all three nlg tasks are in english, with the exception of the input for the mt task, which is in russian for the model-agnostic task and in many other languages for the model-aware task @xcite . ## dataset the dataset for the shroom challenge comprises a compilation of model-generated text entries with the aim to classify each output as either a hallucination of the generative model or not. information for the data sample includes the following fields: (i) src -the input text given to the generative language model; (ii) hyp -the generated textual output of the model; and (iii) tgt -the intended reference or the ground truth text that the model is supposed to generate; (iv) task -the task being solved; (v) labels -five labels, either "hallucination" or "not hallucination" labeled by five annotators, and finally, (vi) p(hallucination) indicates the proportion of annotators that labeled the data sample as a hallucination. the dataset was split in the following way: training data of 30,000 samples without annotations with 10,000 samples for each task; validation data of 499 labeled samples with 187, 187, and 125 samples for dm, mt, and pg tasks, respectively; and test data of 1,500 examples without annotations to evaluate and rank the results of the competitors with 563, 562, and 375 examples for dm, mt, and pg tasks, respectively. validation data sample is presented in all participants' submissions were evaluated using two criteria: ## methodology nli task determines whether a hypothesis follows a premise and classifies it as either entailment, contradiction, or neutral. previous research showed that nli can be successfully used for hallucination detection in summarization @xcite . our system is an ensemble of several nli pretrained transformer-based models. each model defines a new feature set, then a tree-based gradient boosting final estimator is trained on top of these features. it's well known that in-domain training data are crucial for classifier performance @xcite . however, organizers do not provide us with the labeled training set, so we train final estimator on the labeled validation set. the following is a list of nli transformer-based models used in our pipeline: task input output reference p(halluc.) dm because redpillers are usually normies @xmath0. (slang) an incel. (incel, _, slang) a man of a slightly lower ranking on a scale of physical attractiveness than chad. 0.2 mt drn baton! nuclear baton! blimey! 1.0 pg are we resolved? so, what, we're in the clear now? do we have a deal? 0.4 in addition to the output of the before-mentioned models, we add as features the lengths of input, output and reference texts. then we train cat-boost @xcite models as metamodels on top of these features. besides catboost model, we also train random forest @xcite implemented in scikit-learn library @xcite and lightgbm @xcite . catboost yields the best results among them. ## experimental setup we do not use any preprocessing of input texts (premises and hypotheses). neither do we use an unlabeled training set. so, the transformer-based models serve to obtain features from the validation and test sets, then the catboost metamodel is trained on the validation set and predicts the test set. as for the catboost metamodel, we performed the following steps: ## results the results on the test set for both model-agnostic and model-aware tracks are presented in our approach for model-agnostic case provided us with an accuracy of 82.1% and spearman correlation of 0.752. with this approach, our team achieved the 6th place out of 41 in the competition for model-agnostic track. only two teams achieved a higher spearman correlation. the same approach was applied for the modelaware track and provided us with an accuracy of 79.9%, which is the 8th place out of 38 in the competition. the value of spearman correlation turned out to be 0.713. more detailed results of the competition can be found in @xcite . ## conclusion in this paper, we describe the ensemble system for hallucination detection by using transformerbased models. we present a simple, yet effective ensemble pipeline that provided us with results comparable with the best scores for the both tracks. future work might include thoughtful error analysis. improved quality can be achieved by annotating unlabeled training set with llms @xcite . in addition, a multilingual setup of nli models can be used to develop multilingual hallucination detection system @xcite . the proposed approach can be used standalone or can be integrated into the deeppavlov framework @xcite .