NU - RU at S em E val-2024 Task 6: Hallucination and Related Observable Overgeneration Mistake Detection Using Hypothesis-Target Similarity and S elf C heck GPT (2024)

Abstract: One of the key challenges in Natural Language Generation (NLG) is “hallucination,” in which the generated output appears fluent and grammatically sound but may contain incorrect information. To address this challenge, “SemEval-2024 Task 6 - SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes” is introduced. This task focuses on detecting overgeneration hallucinations in texts generated from Large Language Models for various NLG tasks. To tackle this task, this paper proposes two methods: (1) hypothesis-target similarity, which measures text similarity between a generated text (hypothesis) and an intended reference text (target), and (2) a SelfCheckGPT-based method to assess hallucinations via predefined prompts designed for different NLG tasks. Experiments were conducted on the dataset provided in this task. The results show that both of the proposed methods can effectively detect hallucinations in LLM-generated texts with a possibility for improvement.

## introduction natural language generation (nlg) is a field within natural language processing (nlp) that focuses on enabling machines to produce human-like texts. in nlg, one of the challenges is the phenomenon of "hallucination", where the generated output is fluent and grammatically sound but contains incorrect information or extends beyond the provided information. this issue is particularly significant in nlg applications where correctness is crucial, such as machine translation and paraphrasing. it can compromise the quality and reliability of the generated content, resulting in a loss of fidelity to the sources or models from which the content is generated. to address this challenge, "semeval-2024 task 6 -shroom, a shared-task on hallucinations and related observable overgeneration mistakes" @xcite is introduced. this task aims to identify grammatically correct outputs that contain incorrect semantic information or overgenerated content, with or without access to the model that produced the output. the outputs are obtained from various large language models (llms) in three distinct nlg tasks: definition modeling (dm), machine translation (mt), and paraphrase generation (pg). recent efforts have been made to develop frameworks for detecting hallucinations in llmgenerated texts. one approach involves calculating information overlap and contradictions between generated and reference texts @xcite . higher mismatches suggest a greater likelihood of hallucination. another popular approach is an llm-based evaluation. this approach focuses on prompting llms to assess a machine-generated text and determine the probability of this text being a hallucination @xcite . despite the success of these existing methods, they have mainly focused on detecting factual hallucinations. this paper further explores how information overlap calculation and llm-based evaluation approaches can be applied to detect overgeneration hallucinations. specifically, we propose two methods to detect overgeneration hallucinations in se-meval task 6. the first method is hypothesis-target similarity, which measures text similarity between a generated text (hypothesis) and an intended reference text (target). the second method is an llmbased evaluation approach that utilizes a state-ofthe-art framework called selfcheckgpt @xcite . this method assesses hallucinations via distinct predefined prompts tailored for texts generated from different nlg tasks. ## related work recently there have been some attempts to develop frameworks for evaluating hallucinations in llm-1 253 generated texts. one approach is to consider lexical features of llm-generated texts and reference texts and calculate the information overlap and contradictions between the generated and the reference texts. the higher the mismatch counts, the lower the faithfulness and thus the higher the hallucination score. for example, @xcite proposed par-ent (precision and recall of entailed n-grams from the table), which is capable of assessing hallucinations by referencing both the source and target texts. @xcite introduced a metric for knowledge-grounded dialogue tasks aimed at measuring the alignment between llm-generated texts and the relevant knowledge judged by humans. @xcite proposed the bag-of-vectors sentence similarity (bvss) metric for assessing sentence adequacy in machine translation. this metric aids in identifying disparities in information between the output and the translation reference. despite the simplicity and effectiveness of information overlapping, it has limitations in handling syntactic or semantic variations, which can impact its accuracy in evaluating faithfulness. another recent approach is an llm-based evaluation where an llm is prompted to evaluate generated texts, e.g., to predict the probability that a generated text is a hallucination. for instance, @xcite used llms to evaluate the validity of their own claims by asking models to first generate answers and then to evaluate the probability that their answers are correct. @xcite proposed an approach called selfcheckgpt with prompts. in their approach, each llm-generated sentence was compared against multiple generated responses from an llm. an llm was asked to assess whether an llm-generated sentence was supported by the generated responses. if it was consistently supported by multiple responses, then it was likely to not be a hallucination. @xcite proposed the chainpoll approach where an llm was asked to decide whether an llm-generated text contained hallucinations, using a detailed and carefully engineered prompt. however, the majority of existing approaches have primarily focused on detecting factual hallucinations related to incorrect information in texts, rather than overgeneration hallucinations. thus, there remains a critical need to explore and adapt these approaches for the detection of overgeneration hallucinations. ## problem formulation the objective of this task is to predict whether the actual model production (generated text) is a hallucination, with or without having access to the model that generated the text. specifically, each input in this task consists of the following information: for each input, the goal is to predict a label indicating whether the hypothesis is a hallucination, along with the probability of the hypothesis being a hallucination (p(hallucination)). in this task, two datasets were provided: modelaware dataset and model-agnostic dataset. in the model-aware dataset, model checkpoints (available on huggingface) were provided for every sample. conversely, in the model-agnostic dataset, these checkpoints were not included. for each dataset, an unlabeled training set, a validation set (with true labels), and a test set were provided. also, a trial set was given without categorizing the samples based on whether they were model-aware or model-agnostic. the validation, trial and test sets contain binary annotations provided by at least five different annotators, along with a majority vote gold label. ## methods to achieve the task of detecting overgeneration hallucinations, we propose two methods: (1) hypothesis-target similarity and (2) selfcheckgptbased methods. the details of each approach are discussed in the following subsections. ## experiments the experiments were conducted on both modelaware and model-agnostic datasets. for each of these datasets, we applied the proposed methods to both the validation and test sets to evaluate their performance. two evaluation metrics were employed: the accuracy of binary classification and the spearman correlation of the predicted probabilities (p(hallucination)) with the proportion of the annotators marking the hypothesis as "hallucination". we compared the proposed methods with the baseline provided by the task organizers on the test set. this baseline is based on the selfcheckgpt with prompt approach, employing an open-source mistral model @xcite . in this baseline, for samples from a pg task, only the source text is provided as context to the mistral model, similar to our selfcheckgpt-based method. for dm and mt tasks, the baseline utilizes only the target text as context. in contrast, our selfcheckgpt method incorporates both source and target texts as context. also, in this baseline, each prompt was run through the mistral model only once. ## conclusions this work proposes two methods for detecting hallucinations and observable overgeneration mistakes in texts generated by llms. the first method, the hypothesis-target similarity method, involves calculating the information overlap between a generated text and a reference text. this method utilizes a pre-trained sentencetransformer model to calculate text embeddings for both the generated and reference texts, and cosine similarity to measure their similarity. the second method employs an llm-based evaluation approach. it uses the selfcheckgpt technique with prompts tailored to llm-generated texts from various nlg tasks. the experimental results highlight the effectiveness of the proposed hypothesis-target similarity method in detecting hallucinations, particularly 6 258 when the similarity threshold is carefully chosen. additionally, the findings reveal that the proposed selfcheckgpt-based method outperformed the baseline, and effectively identified hallucinations in texts generated by llms. moreover, these results underscore the significance of prompt design in evaluating hallucinations using llms. however, there is still room for improvement in the performance of our methods. for future work, other sentencetransformers models, such as multi-qa or msmarco passage models (sbert.net, 2022) or alternative embedding models, such as infersent @xcite or universal sentence encoder @xcite for the hypothesis-target similarity approach will be considered. as for the selfcheckgpt-based approach, other llms besides gpt-3.5 will also be investigated. moreover, various prompt formats and the use of few-shot examples in the prompt will be explored.