S em E val-2024 Task 6: SHROOM , a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes (2024)

Abstract: This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations: outputs from natural language generation (NLG) systems that are fluent, yet inaccurate. Such cases of overgeneration put in jeopardy many NLG applications, where correctness is often mission-critical. The shared task was conducted with a newly constructed dataset of 4000 model outputs labeled by 5 annotators each, spanning 3 NLP tasks: machine translation, paraphrase generation and definition modeling.The shared task was tackled by a total of 58 different users grouped in 42 teams, out of which 26 elected to write a system description paper; collectively, they submitted over 300 prediction sets on both tracks of the shared task. We observe a number of key trends in how this approach was tackled—many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.

## introduction the modern nlg landscape is plagued by two interlinked problems: on the one hand, our current neural models have a propensity to produce inaccurate but fluent outputs; on the other hand, our metrics are most apt at describing fluency, rather than correctness. this leads neural networks to "hallucinate", i.e., produce fluent but incorrect outputs that we currently struggle to detect automatically. for instance, @xcite report that when trying to produce a paraphrase for the input "i am not sure where my phone is", they obtain the following 'hallucination' behavior: "how can i find the location of any android mobile". for many nlg applications, the correctness of an output is however mission critical. for instance, producing a plausible-sounding translation that is inconsistent with the source text puts in jeopardy the usefulness of a machine translation pipeline. this motivates us to organize a sharedtask on hallucinations and related observable overgeneration mistakes, or shroom. with our shared task, we hope to foster the growing interest in this topic in the community (e.g., @xcite @xcite @xcite . in particular, in the shroom we adopt a post hoc setting, where models have already been trained and outputs already produced. participants were asked to perform binary classification to identify cases of fluent overgeneration hallucinations in two different setups: model-aware and model-agnostic tracks. that is, participants had to detect grammatically sound outputs which contain incorrect or unsupported semantic information, inconsistent with the reference input, with or without having access to the model that produced the output. to that end, we constructed a dataset comprising a collection of checkpoints, inputs, references and outputs of systems covering three different nlg tasks: definition modeling (dm, @xcite , machine translation (mt) and paraphrase generation (pg) trained with varying degrees of accuracy. datapoints were all annotated by 5 human annotators each resulting in 1000 validation items and 3000 test items. beyond simply detecting factually unsupported outputs, one of the goals of this shared task was to establish whether hallucinations are best construed as a categorical phenomenon or a gradient one. similar remarks have been made with respect to textual entailment @xcite . as such, participants' submission were scored both for accuracy (whether classifiers correctly identify hallucinations) and calibration (whether classifiers are confident about their prediction when they ought to be). the shared task attracted a total of 58 different users grouped in 42 teams, out of which 26 elected to write a system description paper. collectively, over the three weeks of the evaluation phase, participants submitted 300 valid sets of predictions on the model-aware track, and 320 on the model-agnostic track. we take this participation rate, along with the breadth of methodological approaches developed by participants, as clear signs of success for our shared task: this large pool of participants allows us to identify and discuss some key trends in how the task was tackled. crucially, many participants rely on a handful of model, and often rely either on synthetic data for fine-tuning or zero-shot prompting strategies. in terms of raw performance, we note that while a majority of the teams (64 to 71%) did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items. in sum, this first iteration of the shroom underscores both an interest of the research community as well as the current limitations in our approaches. the remainder of this article is structured as follows: iin section 2, we provide an overview of the current research landscape. section 3 defines our theoretical framework, and section 4 summarizes our data collection process. we then present and discuss shared task results in sections 5 and 6 before concluding with a few thoughts on further research in section 7. it is now widely accepted that nlg models often generate outputs that are not faithful to the given input, commonly referred to in the community as hallucinations @xcite @xcite ). yet there is minimal consensus on the optimal framework for its application. this lack of agreement is due in part to the diversity of tasks that nlg encompasses @xcite . @xcite propose a taxonomy of hallucinations that includes oscillatory productions, and fluent but strongly or fully "detached" outputs. while this taxonomy is well constructed, we find it inadequate for the needs of the community at large for four reasons: (i) it conflates some issues of fluency with semantic correctness (oscillatory productions are cases of non-fluent overgeneration where no extraneous semantic material is introduced); (ii) it only considers the most extreme cases of hallucinations (strongly or fully detached productions), whereas diagnosis of intermediary cases is bound to be more challenging and useful to the community; (iii) it focuses only on mt, although other tasks are also known to suffer from fluent overgeneration (e.g., @xcite , including the ones we propose to address; (iv) it uses only lowest scoring outputs, whereas any tool built to verify system outputs ought not to flag nonpathological outputs. alternative studies have built benchmarks for hallucination detection, with a predominant emphasis on dialogue systems. @xcite propose the halueval benchmark using an annotation framework that does not necessarily center on the input given to the model and requires the annotators to search the internet for facts. moreover, they opted to annotate the outputs of a popular llm, with the major downsides that it is closed, not-transparent and commercial; rendering the research outputs that may stem from future studies less interesting. other benchmarks include the works of @xcite and @xcite , which automatically insert hallucinations into training instances to generate syntactic data for token-level hallucination detection; @xcite , which work with factual claims supported by reliable, publicly available evidence; and @xcite , which focus on knowledge-based dialogue systems and base their annotation on nli, relying only on the system's input, just as we do. ## tripping over hallucinations: task definition and annotation in contrast with previous works (e.g. @xcite , we focus on cases of fluent overgeneration since judgments pertaining to the over-generative nature of a production can be elicited by means of inferential semantics: if an output cannot be inferred from its semantic reference, then it contains some information that is not present in the reference-i.e., the model has generated more than we expected. 1 this approach connects with the theoretical framework sketched by van deemter (2024), who likewise relies on inferential semantics but also considers undergeneration issues in nlg outputs. we provide multiple annotations and a gold majority label, given the low consensus on semantic annotations @xcite . in figure 2 we provide an overview of the task. the shroom is framed around two key distinctions: (i) model-aware vs. model-agnostic approaches, and (ii) source-referential vs. dualreferential datapoints. the former corresponds to whether participants have access to the model that generated the item: model-agnostic approaches are practical, as models may not be accessible to end users; model-aware approaches can lead to richer and more accurate diagnoses. the latter is a consequence of our inferential take on over-{ "hyp":"a cigarette .", "ref":"tgt", "src":"i stepped outside to smoke myself a j . what is the meaning of j ?", "tgt":"( plural js or j 's ) a marijuana cigarette .", "model":"ltg\/flan-t5-definition-en-base", "task":"dm", "labels":["hallucination","not hallucination", "not hallucination", "hallucination", "hallucination"], "label":"hallucination", "p(hallucination)":0.6 } generation: what can effectively serve as a semantic reference varies across nlp systems. for dm, where we fine-tune a language model to produce a definition for a given example of usage the datapoints are target-referential, i.e. the target is the sole usable semantic reference. in this context, the target serves as the sole usable semantic reference. conversely, the target is expected to be semantically implied from the source in source-referential tasks, such as summarization. note that we do not annotate source-referential tasks due to annotation challenges that make them unreliable for our purposes. in dual-referential tasks like pg & mt, this distinction bears no weight. in figure 3 , we present an example datapoint displaying how we plan to encode all relevant information in a json format is provided. the datapoint keeps track of the source provided to the model as input (src), the intended target (tgt), the model production (hyp), the task this production was derived from (task), can correspond to dm, mt or pg), whether this datapoint is targetreferential (ref), the annotations, the gold label and the proportion of annotators that labeled the utterance as a hallucination (labels, label, and p(hallucination)). in the model-aware track, we will also provide a huggingface model name (model). ## foraging and harvesting season: collected data all shroom data (models, outputs and annotations) are available under a cc-by license. foot_1 ## a bunch of fun guys: qualitative analysis of participants systems we derive our analyses from system description papers as well as self-reports from a handful of participants who elected to not provide a full description of their systems. this corresponds to 33 systems out of the 42 identified teams that participated to the shared task, out of which 7 did not provide a full description. see also how the task was approached. the teams used a variety of methods to address the problem, ranging from ensemble techniques to fine-tuning pretrained language models (llms) and prompt engineering. as expected, most teams used popular pretrained llms such as gpt, llama, deberta, roberta, and xlm-roberta; figure 6 provides a summary of which models were most popular among our teams. the vectara hallucination evaluation model foot_9 turned out to be extremely popular, as more than 1 in 4 teams that provided information about their systems report having used it in their experiments. if we add other deberta-based models, this number climbs to 16/33, i.e. almost every other team used deberta or a variant thereof. yet, the ways in which these llms were used cover a wide range of approaches: some either fine-tuned on hallucination data or optimized with prompts; others employed in-context learning with role-playing, automatic prompt generation, and ensemble methods. furthermore, some teams focused on zero-shot and few-shot approaches, while others focused on synthetic data generation and semisupervised learning techniques to construct a labeled training set. especially noteworthy, @xcite report constructing a manual dataset of 3000 datapoints for training their systems. teams predominantly relied on the data constructed for the shroom, although some teams added datasets such as qqp and paws. interestingly, we also note five teams relying on nli/entailment data or models, including some that achieved high results @xcite @xcite and team centre-back)-and this matches the theoretical framework adopted in this shared task. what worked well. we now turn to what distinguishes top scorers from other submissions. we note that systems based on the closed-source models gpt-3.5 and gpt-4 tend to fare well: 4 out of the 6 highest scoring systems on either track- @xcite ; @xcite ; @xcite ; @xcite and alejandro mosquera-all report using these models. this is however not a strict requisite as opdai @xcite manages to rank high (2 nd on the modelagnostic track and 4 th on the model-aware track) without it. neither does using closed-source models guarantee a high result: ucc-nlp and markchom et al. ( 2024 ) also use gpt-3.5, and while the former is ranked 14 th on the model-agnostic track and 13 th on the model-aware track, the latter is ranked 30 th on the model-agnostic track and 21 st on the model-aware track, and only outperforms the baseline model in accuracy by 0.02 to 0.03 points. remarkably, many of the top-scoring approaches rely on fine-tuning @xcite @xcite or ensembling @xcite , suggesting that high performances do not come out of the box from off-the-shelf llms and systems. it is necessary to adapt existing models or establish to what extent their predictions are useful to the task at hand. another important trend we identify is that the number of submissions per team anti-correlates with the rank they obtain: the more participants submitted, the higher their best scores went. this is visualized in figure 7 : on both tracks, we find reasonable anti-correlations (-0.58 < ρ < -0.44) indicating that top-scorers tended to submit more. this might provide an alternative explanation for what distinguishes top-scorers from other participants: if we were to model participants' submissions as a random process, we would expect that sampling more often (i.e., submitting more) would mechanically yield a better rank. overall, the high methodological diversity highlights the complexity of hallucination detection, even when contained the simple inferential semantics framework of our shared task: while a focus on nli or using high-performance closed source models may help, the highest scores are obtained through thorough involvement-both in terms of model training and prediction set submissions. ## much room to grow: conclusions and future perspectives this first iteration of the shroom shared task on detecting hallucinations has allowed us to make significant headway into understanding the confabulatory behavior of modern nlg systems. the data collected demonstrate that hallucinations correspond to a gradient phenomenon, and that different speakers form different opinions as to what counts as a hallucination. we were also able to showcase that ambiguous items remain challenging, and that the current state of the art on the dataset we provided is compatible with simple random guesses whenever the data is more ambiguous. this results underscore the massive gap that nlp research urgently needs to address: one out of every six items is still misclassified by the most effective systems showcased during this shared task. the diversity of methodologies employed by participants underscores how out-of-the-box solutions are not sufficient: highest scoring teams had to rely on fine-tuning or ensembling and made a high number of submissions. relatedly, access to the model parameters was of limited help: few approaches attempted to perform model-specific investigations, and performances on the model-aware track are in fact lower than what we observed on the modelagnostic track. properly leveraging the parameter space for finer-grained hallucination detection remains a point for future research to investigate. this shared task has not broached some crucial aspects and questions: how do these results translate insofar as modern llms-often much larger and better trained than the systems we studied here-are concerned? can we leverage sentencelevel predictions to pinpoint token-level issues with the output of our nlg systems? and will the difficulties that we underscored in this purely english be exacerbated when studying other languagesespecially those that are less well-resourced and typologically different? answering these questions and more will require further research-and perhaps future iterations of this shared task. overall, the success of this shared task is owed to its committed participants. we received over 350 submissions in the span of three weeks from across the world. the width of approaches studied and reported upon provides a useful snapshot of where the field is at, what approaches are favored, and what gaps still need to be overcome. we expect that the results of the shroom will provide a useful starting point for future work on hallucinations.