IRIT -Berger-Levrault at S em E val-2024: How Sensitive Sentence Embeddings are to Hallucinations? (2024)

Abstract: This article presents our participation to Task 6 of SemEval-2024, named SHROOM (a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes), which aims at detecting hallucinations. We propose two types of approaches for the task: the first one is based on sentence embeddings and cosine similarity metric, and the second one uses LLMs (Large Language Model). We found that LLMs fail to improve the performance achieved by embedding generation models. The latter outperform the baseline provided by the organizers, and our best system achieves 78% accuracy.

## introduction in recent years, with the emergence of the foundation models, the generation of hallucinated text has become an increasingly prominent and alarming issue. despite the state-of-the-art performances achieved by the latest text generation models, such as gpt-4 @xcite or llama 2 @xcite , the problem of hallucinations remains open, making these models challenging to apply in real-world applications. hallucination is defined as a segment of text that appears fluent and natural but contains incoherent and inconsistent information compared to the provided context @xcite . the problem of hallucinations appears in several nlg tasks such as text summarization @xcite and machine translation @xcite . the shared-task shroom foot_0 falls within the scope of these tasks. the aim of shroom is to identify samples containing hallucinations with regard to the provided context through a binary classification. the task is established in a post hoc setting, where models have already been trained to generate text based on the provided context. three text generation tasks are considered: machine translation (mt), paraphrasing generation (pg), and definition modeling (dm). one can find in source : j'ai poli le plancher et les meubles. target : i polished up the floor and furniture. hypothesis : i've got the floor and the furniture. ref : either labels : [hallucination, hallucination, hallucination] label : hallucination p(hallucination) : 1.0 the binary classification can be performed in two different tracks: the model-aware and modelagnostic tracks. in the model-aware track, the checkpoint of the model that generated the hypothesis is provided and can be used in the classification system, which is not the case for the modelagnostic track. for more information on task 6 of semeval-2024 shroom as described by its organizers, we invite the reader to consult the paper by @xcite . in the literature, detecting hallucinations in sentences mainly relies on comparing segments of text. among these approaches, one can cite those based on named entities @xcite : the idea is to determine if the generated text contains incon-sistent entities compared to the source. other approaches are based on question-answering methods, where the aim is to answer a set of questions and evaluate the difference between the two texts @xcite . however, data provided for the shroom task are of a particular nature as it consists of very short texts, composed of one or two concise sentences. this characteristic limits the use of several detection methods, such as those based on named entities, since they are very rare in these texts. the same goes for question-answering methods. this observation has led us to turn to simpler methods, involving capturing the semantics of sentences in a general way. in this paper, we present the two lines of approaches we investigated for the task: we solve the hallucination classification task using the cosine similarity metric between sentence embeddings of the reference and hypothesis (see section 4.1), our sentence embedding approaches outperformed our llm ones. the former have proven to be very relevant given the shortness of the sentences and the low risk of losing information. our best performing system obtained the accuracy of 78% in both tracks which puts us in position 22/48 in the model-agnostic track and 22/45 in the modelaware track. ## related work approaches in the literature can be classified depending on the hallucinatory content to detect @xcite . some works focus on the detection and comparison of existing entities between the context and the generated hypothesis, such as @xcite . they are based on the assumption that human brain is sensitive to different types of information, such as named entities and proper nouns when reading, and mistakes concerning named entities are striking to human users @xcite . @xcite go further by evaluating facts (entities and relations). another line of works focuses on the use of question-answering as an indi-cator to identify hallucinations @xcite , or the use of text entailment, which consists in determining whether the generated text is a hallucination if it cannot be entailed by the source @xcite . other approaches focused on the classification of hallucination types (whether they are intrinsic or extrinsic @xcite ), or factual or non-factual @xcite . large language models have also been used to determine whether the generated text contains hallucinatory content. the aim of these methods is to set up prompts to compare the sources and the hypotheses @xcite . other methods make specific prompts to ask the llm to "think" and judge whether a given text contains hallucinations, justifying its answer by producing a chain-of-thought (cot) explanation @xcite . in this paper, we explore a simpler method that consists in calculating the embeddings of the hypothesis and the reference, and computing their semantic similarity using the cosine similarity metric. contextual embeddings have been successfully used in various nlp tasks, such as sentiment analysis @xcite or topic modeling @xcite . our underlying idea here is to see how sensitive the semantic similarity is to the hallucinated content in sentences and to what extent the cosine similarity metric reflects this sensibility. ## data description the organizers of shroom provided data from 3 different nlg tasks : machine translation, paraphrasing generation, definition modeling. each task is divided into two tracks: model-aware track and model-agnostic track. we were provided 5 datasets in the development phase : train-aware, train-agnostic, trial, dev-aware and dev-agnostic, containing 30000, 30000, 80, 501 and 499 samples respectively and 2 different test datasets in the evaluation phase: test model-aware and test model-agnostic, containing each 1500 samples. for each sample, the model-generated hypothesis was annotated by 3 (trial dataset) or 5 different annotators (dev and test datasets) (labels in ## system overview as the training dataset provided by organizers is not labeled, we decided to experiment unsupervised approaches, either using sentence embeddings or llms.