Compos Mentis at S em E val2024 Task6: A Multi-Faceted Role-based Large Language Model Ensemble to Detect Hallucination (2024)

Abstract: Hallucinations in large language models (LLMs), where they generate fluent but factually incorrect outputs, pose challenges for applications requiring strict truthfulness. This work proposes a multi-faceted approach to detect such hallucinations across various language tasks. We leverage automatic data annotation using a proprietary LLM, fine-tuning of the Mistral-7B-instruct-v0.2 model on annotated and benchmark data, role-based and rationale-based prompting strategies, and an ensemble method combining different model outputs through majority voting. This comprehensive framework aims to improve the robustness and reliability of hallucination detection for LLM generations.

## introduction the modern natural language generation (nlg) @xcite landscape faces two interconnected challenges: firstly, current neural models have a tendency to produce fluent yet inaccurate outputs, and secondly, our evaluation metrics are better suited for assessing fluency rather than correctness @xcite . this phenomenon, known as "hallucination," @xcite where neural networks generate plausible-sounding but factually incorrect outputs, is a significant hurdle, especially for nlg applications that require strict adherence to correctness. for instance, in machine translation @xcite , producing a fluent translation that deviates from the source text's meaning renders the entire translation pipeline unreliable. this issue may arise as llms are trained on vast amounts of data from the internet, which can contain inaccuracies, biases, and false information. also, it may arise due improper representations learned during training even if good quality data is used. as a result, llms can sometimes hallucinate or fabricate details, especially when prompted to discuss topics outside their training data or make inferences beyond their capabilities. hallucination detection @xcite , also known as factual verification or truthfulness evaluation, identifies and mitigates these hallucinations in the outputs of llms. this is an active area of research and development, as it is crucial for ensuring the reliability and trustworthiness of llmgenerated content, particularly in high-stakes domains such as healthcare, finance, and legal applications. in this task, the primary focus will be to classify whether a generation is hallucinated. this work proposes a multi-faceted approach to detecting hallucinations in large language models' outputs. we employ automatic data annotation using a proprietary llm (claude 2.1 2 ) to label examples from the provided training set as hallucinated or not. then we fine-tune the mistral-7b-instruct-v0.2 3 model on this annotated data as well as the halueval benchmark @xcite to create two fine-tuned models. to improve performance, we use role-based prompting that casts the task in specific contexts like fact-checking. we also leverage rationale-based prompting, asking the llm to justify its hallucination label. finally, an ensemble method combines outputs from the fine-tuned mistral models, claude 2.1, and different prompting strategies via majority voting. this comprehensive approach aims to enhance the robustness and reliability of hallucination detection across various language tasks. dressing this issue. participants are tasked with performing binary classification to identify instances of fluent overgeneration hallucinations in two different setups: a model-aware track and a modelagnostic track. essentially, participants must detect grammatically sound outputs that contain incorrect or unsupported semantic information, inconsistent with the source input, with or without having access to the model that produced the output. to facilitate this task, participants are provided with a collection of checkpoints, inputs, references, and outputs from systems covering three different nlg tasks: definition modeling (dm), machine translation (mt), and paraphrase generation (pg). these systems will be trained with varying degrees of accuracy. the validation and test sets will include binary annotations from at least five annotators, with a majority vote determining the gold label. ## results ## conclusion this work proposes a multi-faceted approach for detecting hallucinations in large language model outputs across various natural language tasks. it employs automatic data annotation, fine-tuning stateof-the-art models on annotated data and benchmarks, role-based and rationale-based prompting strategies, and an ensemble method combining multiple model outputs. the ensemble pipeline achieves promising results on model-agnostic and model-aware evaluation settings for hallucination detection. while challenges remain, this comprehensive framework highlights the potential of carefully designed prompting, model fine-tuning, and ensembling techniques to enhance the robustness and reliability of factual verification in language model generations, paving the way for developing more trustworthy natural language generation systems.