SLPL SHROOM at S em E val2024 Task 06 : A comprehensive study on models ability to detect hallucination (2024)

Abstract: Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledgeor the source text. This study explores methodsfor detecting hallucinations in three SemEval2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation.We evaluate two methods: semantic similaritybetween the generated text and factual references, and an ensemble of language modelsthat judge each other’s outputs. Our resultsshow that semantic similarity achieves moderate accuracy and correlation scores in trial data,while the ensemble method offers insights intothe complexities of hallucination detection butfalls short of expectations. This work highlights the challenges of hallucination detectionand underscores the need for further researchin this critical area.

## introduction while natural language generation (nlg) has empowered machines to craft increasingly sophisticated text, transforming the nlp landscape, a dark undercurrent lingers -the phenomenon of hallucinations. in nlg, hallucinations refer to fabricated or misleading content woven into a generated text, deviating sharply from reality @xcite . these fictional elements, despite seeming plausible due to their learned patterns, threaten the very core of nlg's promise: reliability and truthfulness @xcite . imagine summarizing news articles riddled with fictional details or translating medical instructions brimming with inaccuracies. such scenarios underscore the profound and potentially dangerous implications of hallucinations within nlg, making their detection and mitigation an urgent priority @xcite . the specter of hallucinations looms large over nlg, particularly in domains demanding unyield-ing accuracy and safety. imagine a medical summary riddled with invented details or medication instructions marred by mistranslations -these scenarios, chillingly possible, could directly jeopardize patient well-being @xcite . recognizing this critical threat, researchers have embarked on a mission to untangle the complexities of hallucinations, developing methods for their detection and ultimately, prevention @xcite . this paper dives headfirst into the challenge of hallucination detection within nlg, leveraging recent advancements in the field. we employ diverse methodologies to unmask these fictional elements in semeval 2024 task 6 evaluation data. firstly, we assess the semantic similarity between generated text (hypotheses) and the provided reference outputs, gauging their alignment in meaning. secondly, we harness the power of cosine similarity of embeddings, allowing us to capture subtle semantic nuances and relationships within text representations. furthermore, we integrate natural language inference (nli), analyzing whether the generated text logically implies or contradicts factual information. additionally, we utilize large language models (llms) to discern context similarity, leveraging their inherent language understanding to identify inconsistencies that might point toward hallucinations. but our approach goes beyond established techniques. we introduce a novel judgment llm framework, where one llm acts as a discerning judge, scrutinizing the outputs of other llms for signs of hallucination. this innovative approach leverages the collective strengths of multiple models while introducing an element of meta-reasoning to the detection process. knowledge-based approaches: fact verification compares generated text with information from a domain-specific knowledge base. this approach can be effective but requires substantial knowledge bases and may not generalize well to unseen domains. classification approach: @xcite created a dataset specifically for hallucination detection, but it has not been very successful. white-box and grey-box approaches: hidden state analysis: @xcite ) use an mlp classifier on the llm's hidden states to predict truthfulness. this requires access to internal model states and may not apply to all architectures. token probabilities: grey-box methods analyze the token probabilities generated by the llm, assuming factual sentences contain high probability tokens. however, this can be unreliable for complex or ambiguous phrases. black-box approaches: self-evaluation: @xcite propose asking the llm itself to assess the likelihood of its output being correct. while promising, this method relies on the llm's self-awareness and may not be reliable for all models. proxy model: this approach uses a publicly available llm to estimate the token probabilities of the black-box model's output and infer its factual consistency. however, its accuracy depends on the proxy model's similarity to the black-box model. selfcheckgpt @xcite introduced a black box approach. the main idea of this study is that if the llm is trained on a concept if multiple responses are taken from it, the samples will be similar and include consistent facts. whereas if it is hallucinating, the samples will be different and contradictory. therefore, several samples are taken from the llm, and by measuring the information consistency between the responses, we can understand whether they are factual or hallucinated. our approach: this paper builds upon existing work by combining elements from different categories. we leverage information consistency within multiple llm responses, inspired by selfcheckgpt @xcite , but introduce a novel "judgment llm" framework that goes beyond selfevaluation by employing one llm to scrutinize the outputs of others. this approach aims to address previous methods' limitations by leveraging mul-tiple models' collective strengths and introducing meta-reasoning into the detection process. ## task description the shroom challenge shines a light on a crucial hurdle in natural language generation (nlg) -pinpointing seemingly correct text that holds inaccurate meaning, often referred to as "misleading outputs." we, alongside other participants, are tasked with detecting these "semantic hallucinations," even when they are flawlessly written and grammatically sound. the challenge focuses on "fluent overgeneration," where generated text, despite being linguistically coherent, strays from the intended semantic meaning. participants operate in a "post hoc" setting, assuming models have already been trained and their outputs generated. this is where we step in -to identify these misleading texts amidst seemingly accurate ones. this is critical to ensure the truthfulness and reliability of nlg outputs, especially in real-world applications. the shroom challenge presents a twopronged approach to tackle fluent overgeneration hallucinations: model-aware and model-agnostic tracks. participants can choose to leverage knowledge of the model or not, depending on their approach. this multifaceted assessment covers three key nlg domains: definition modeling, machine translation, and paraphrase generation. the challenge provides a rich dataset including: ## proposed systems in our study, we tried to do the hallucination detection task in two separate methods and tried to compare and analyze their results: ## results and analysis semantic similarity and nli: the deberta model or nli method outperforms all other methods, suggesting that incorporating natural language inference strengthens our ability to discern hallucinations by capturing semantic relationships between generated text and reference data. ensemble llms and the judgment method: furthermore, our study explored the effectiveness of ensemble llms utilizing a judgment method. surprisingly, the results indicate that ensemble llms' performance wasn't superior to the previous method; in fact, it was even lower. zephyr, acting as a judge, exhibited lower accuracy and correlation scores compared to zephyr alone. while mistral, in the role of a judge, showed improved performance compared to mistral alone, it still falls short of methods like deberta and the labse model, suggesting limitations in this approach's effectiveness for hallucination detection. ## conclusion this study investigated three distinct methods for hallucination detection in language models: the semantic similarity method, nli and the ensemble llms with judgment method. by analyzing and comparing these approaches, we gained valuable insights into their efficacy and suitability for identifying hallucinatory content in model-generated text.