HIT - MI & T Lab at S em E val-2024 Task 6: D e BERT a-based Entailment Model is a Reliable Hallucination Detector (2024)

Abstract: This paper describes our submission for SemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. We propose four groups of methods for hallucination detection: 1) Entailment Recognition; 2) Similarity Search; 3) Factuality Verification; 4) Confidence Estimation. The four methods rely on either the semantic relationship between the hypothesis and its source (target) or on the model-aware features during decoding. We participated in both the model-agnostic and model-aware tracks. Our method’s effectiveness is validated by our high rankings 3rd in the model-agnostic track and 5th in the model-aware track. We have released our code on GitHub.

## introduction in tasks related to natural language generation, the output of a model may be fluent but may suffer from inaccuracies or inconsistencies with the input, a phenomenon referred to as "hallucination." for instance, @xcite and @xcite noted that in machine translation tasks, translated text is regarded as a "hallucination" when it exhibits a complete disconnect from the source text. such discrepancies can mislead users and potentially lead to severe consequences. however, current evaluation metrics such as perplexity and bleu @xcite ) concentrate more on fluency rather than the accuracy or fidelity to the original input. therefore, hallucination detection poses a big challenge and has gathered attention from research community. semeval-2024 task 6 @xcite ) presents a testbed to evaluate whether the model outputs are hallucinating or not. the task comprises a total of three kinds of subtasks, which are definition modeling (dm) @xcite , machine translation (mt) and paraphrase generation (pg). each subtask involves triplet data with a source, which is the input to the model; a target, which represents the "gold" text that the model is expected to produce; a hypothesis, which is the actual output of the model. for all subtasks, the objective is to evaluate whether the hypothesis exhibits hallucinations according to the source or the target. more specifically, the hallucination of the hypothesis is verified based on target for dm and mt tasks, and source for pg task. this paper presents the participation of hit-mi&t lab in the shared task in detail. we introduce four distinct hallucination detection methods, which transform the problem into different tasks: finally, different groups of methods are ensembled for further enhancement, based on the accessibility of model-aware features. with our proposed framework, we achieved the third position in the model-agnostic track and the fifth position in the model-aware track, validating its effectiveness. ## related work with the success of chatgpt (openai, 2022) and gpt-4 (openai, 2023), natural language generation has gained significant prominence within the broader domain of artificial intelligence. its applicability spans a diverse array of tasks, including machine translation, summarization, and story continuation, etc. however, these models are sometimes prone to generating outputs that are fluent yet factually inaccurate, a phenomenon referred to as "hallucination". this phenomenon poses a substantial challenge to the reliability of language generation in real-world scenarios. in the domain of hallucination detection methods, there has been considerable work by predecessors. some people rely on semantic similarity measures for detection, such as n-gram-based metrics (rouge @xcite and bleu @xcite ). however, these metrics only evaluate the lexical overlap between generated texts and reference texts by measuring the n-gram cooccurrence, and cannot discern fine-grained contextual semantic mismatch. other studies @xcite @xcite have fine-tuned bert models using entailment datasets. these fine-tuned models are then utilized to detect hallucinations in specific scenarios. however, the fine-tuning process requires annotation data and can not generalize well among different scenarios. as the hallucination mainly comes from the decoding procedure, some people propose to rely on uncertainty measures to detect hallucination. some research @xcite proposed to calculate the log-probability or its entropy of translations for language generation tasks, and a lower probability indicates a lack of confidence, suggesting a potential hallucination. however, access to token-level probability distributions, essential for these approaches, is limited to opensource models and unavailable for models accessed solely through apis, such as gpt-4. recently, with the popularization of llms, several llm-based methods have been proposed. self-checkgpt @xcite , employs a sampling-based strategy, which involves the generation of multiple stochastic samples. this approach hypothesizes that a model with a good understanding of the concept is less likely to generate significant hallucinations. @xcite has explored the examination of self-contradiction within the context generated by an llm as another aspect of hallucination detection. their experiments, which involved prompting the llm to perform a detection task, have demonstrated successful detection across various llms. ## methods in this section, we will introduce our proposed four groups of methods for hallucination detection. the overall framework is shown in figure 1 . ## conclusion in this study, we aimed to address the hallucination detection problem in semeval-2024 task 6. we established an ensemble model that includes entailment recognition, similarity search, and factuality verification models. for the model-aware track, we further leveraged confidence estimation for augmentation. our approach proved effective as we ranked 3rd in the model-agnostic track and 5th in the model-aware track. although several methods were incorporated in our experiments, we realized that the best result was achieved primarily by relying on the debertabased entailment model. given its portability and generalizability, we plan to further explore its use in hallucination detection in the future.