MAR i A at S em E val 2024 Task-6: Hallucination Detection Through LLM s, MNLI , and Cosine similarity (2024)

Abstract: The advent of large language models (LLMs) has revolutionized Natural Language Generation (NLG), offering unmatched text generation capabilities. However, this progress introduces significant challenges, notably hallucinations—semantically incorrect yet fluent outputs. This phenomenon undermines content reliability, as traditional detection systems focus more on fluency than accuracy, posing a risk of misinformation spread.Our study addresses these issues by proposing a unified strategy for detecting hallucinations in neural model-generated text, focusing on the SHROOM task in SemEval 2024. We employ diverse methodologies to identify output divergence from the source content. We utilized Sentence Transformers to measure cosine similarity between source-hypothesis and source-target embeddings, experimented with omitting source content in the cosine similarity computations, and Leveragied LLMs’ In-Context Learning with detailed task prompts as our methodologies. The varying performance of our different approaches across the subtasks underscores the complexity of Natural Language Understanding tasks, highlighting the importance of addressing the nuances of semantic correctness in the era of advanced language models.

## introduction the shroom task @xcite aims to address the challenge of detecting grammatically sound outputs containing incorrect semantic information in nlg systems. this task is crucial due to the prevalent issue of neural models producing fluent but inaccurate outputs, referred to as "hallucinations" @xcite . given the critical importance of correctness in nlg applications, shroom aims to foster interest in automating the detection of these hallucinations. participants were * equal contribution. our system leveraged three distinct approaches to tackle the shroom task: a baseline cosine similarity, multinli classification @xcite , and large language models (llms), specifically mixtral-8x7b-instruct @xcite . each approach was tailored to identify hallucinations in nlg outputs by comparing them with the source input and detecting inconsistencies in semantic information. through various combinations of these approaches, we aimed to accurately identify grammatically sound but incorrect outputs generated by neural models. ## background and related work previous research efforts have attempted to detect and control @xcite hallucinations. @xcite worked on the origins of hallucinations, concluding @xmath0. @xcite proposed a simple extension to beam search to reduce hallucination. obaid ul @xcite proposed a natural language inference (nli) based method to preprocess the training data to reduce hallucinations. the most similar to our work, guerreiro et al. (2023) studied hallucinations in neural machine translation. they analyzed multiple methods to detect hallucinations and developed dehallucinator which overwrites the translation detected as a hallucination with a better one. ## system overview in this section, we provide an overview of the three approaches employed in our system to address the shroom task. ## experimental setup in this section, we outline the experimental setup used for evaluating our system's performance on the shroom task. ## results in this section, we present the quantitative analysis of our system's performance on the shroom task. we evaluated the performance of our system approaches on the test data for each of the three subtasks: paraphrase generation (pg), machine translation (mt), and definition modeling (dm). our system comprises three distinct approaches: cosine similarity, mnli classification, and large language models (llms), specifically mixtral. we note that mixtral is the only system submitted to the task, and other results are post-evaluation experiments. ## conclusion in conclusion, our system for addressing the shroom task employed three distinct approaches: baseline cosine similarity, mnli classification, and mixtral. each approach was carefully designed to tackle the challenge of identifying hallucinations in natural language generation outputs. our experimental results demonstrated varying degrees of success across the different subtasks. while cosine similarity and mnli classification showed promising performance, leveraging llms proved to be particularly effective in accurately identifying hallucinations. looking forward, our system's performance suggests several avenues for future work. firstly, further exploration and refinement of each approach tailored to the specific subtleties of each subtask could potentially yield improved performance. additionally, investigating ensemble methods or hybrid approaches that combine the strengths of different techniques may enhance overall system robustness. despite the challenges encountered, our system's competitive performance in the shroom task underscores the importance of automated, multiexpert, mechanisms for detecting and mitigating hallucinations in nlg systems. as the field continues to evolve, addressing these challenges will be crucial for advancing the reliability and accuracy of nlg applications. in summary, our system represents a significant step towards addressing the complexities of hallucination detection in nlg outputs, and we are optimistic about the potential for future advancements in this area.