Pollice Verso at S em E val-2024 Task 6: The R oman Empire Strikes Back (2024)

Abstract: We present an intuitive approach for hallucination detection in LLM outputs that is modeled after how humans would go about this task. We engage several LLM “experts” to independently assess whether a response is hallucinated. For this we select recent and popular LLMs smaller than 7B parameters. By analyzing the log probabilities for tokens that signal a positive or negative judgment, we can determine the likelihood of hallucination. Additionally, we enhance the performance of our “experts” by automatically refining their prompts using the recently introduced OPRO framework. Furthermore, we ensemble the replies of the different experts in a uniform or weighted manner, which builds a quorum from the expert replies. Overall this leads to accuracy improvements of up to 10.6 p.p. compared to the challenge baseline. We show that a Zephyr 3B model is well suited for the task. Our approach can be applied in the model-agnostic and model-aware subtasks without modification and is flexible and easily extendable to related tasks.

## introduction language models are outstanding, but 2 they can hallucinate, i.e. generate texts that are not supported by the input or the context. hallucinations can undermine the credibility and usefulness of llms, especially for applications that require high accuracy and reliability, such as summarization, question answering, or dialogue. therefore, there is a pressing need for developing methods to detect and mitigate hallucinations in llms, as well as to understand the causes and effects of this phenomenon. in this semeval challenge @xcite , the task is to detect llm hallucinations based † these authors contributed equally to this work. 1 https://en.wikipedia.org/wiki/pollice_verso 2 https://x.com/chrisgpotts/status/ 1686802492104028160 on the input task given to the llm, the llm response, and the ground truth answer. here, the input tasks can be definition modeling (dm), machine translation (mt) or paraphrase generation (pg). each task contains multiple examples that are fed through an llm and its response is classified as hallucination or not by five annotators. there are two subtasks in this challenge: in the model-aware subtask, access to the generating model is given, while in the model-agnostic subtask, the generating model is unknown. we approach both subtasks in the same way, by removing the model information from the model-aware subtask. while we are certain that access to the generating model can be beneficial, we argue that the model-agnostic setting has better transferability in practice. as the competition baseline (which uses self-checkgpt by @xcite ), we frame the task of hallucintion detection as a "consistency checking" problem with the given information, where the goal is to check whether an llm generation is supported by the ground truth. if the generation is not supported by the ground truth, new and thus probably false information must be present; the llm has hallucinated its response. for building an intuition for our approach, we imagine the same setting in the real world: a person responds to a question and our goal is to detect if this is a hallucination, i.e., the response is not supported by the truth. with this real-world setting in mind, we formulate three intuitions that we later transfer to the challenge baseline: trainset ground truth guiding lm (chatgpt3.5) llama2-7b mistral-7b zephyr-3b zephyr-7b phi2-2.7b prompt optimization loop 5 optimized prompts per model devset ground truth identify best ensembles (averaging or logistic regression) our best ensemble across models and prompts zeph 3b p: 3 ze ph 3b p:4 zep h7b p:2 ze ph 7b p: 1 context: … sentence: … averaged verdict across logits: "hallucination" vs. "not hallucination" figure 1 : overview of our approach. we automatically optimize the prompts of multiple "expert" llms to check the consistency of the given model output with the ground truth. we combine multiple experts (models-promptcombinations) in a uniform or weighted manner and select the best ensemble on an internal validation set. this ensemble performs the final collective verdict, as illustrated by a part of jean-leon gerome's painting "pollice verso" (1872), which represents the ancient roman gesture for judgment on defeated gladiators 1 . ## system description given this intuition regarding a potential approach to the task in the real world, we now connect these steps to our submitted system. an overview is found in figure 1 . ## results we overall have three dimensions in which we can select the best model/prompt ensemble for challenge submission: the resulting model selections as well as the organizer's baseline with their metrics for our validation and test datasets as well as the official task results are shown in first, the last two rows of the table show that our proposed output generation method mostly improves the baseline scores slightly. second, all of our model ensembles are better than the organizer's baseline, showing that the ensemble and logistic regression implementations help in this setting. note that sometimes, the best model ensemble for different dimensions is the same, e.g. zephyr 3b p:4 is the best model when evaluated on the validation correlation, regardless of the ensemble strategy. third, nearly all selected model ensembles contain the zephyr 3b model in some form, which is surprising, as the most other models have more than twice the parameters. this shows that in this task, the model size does not correlate with performance. overall, both zephyr (3b and 7b) models are well-suited for the task, even though they are not trained by the same companies and thus not directly related. in terms of official task results, the model ensemble chosen by the best test accuracy (and then by correlation) shows the best accuracy of our submissions on the model-agnostic task (80.3%). it is an ensemble consisting of two versions of the zephyr 3b model as well as three versions of the zephyr 7b model, combined through a logistic regression. it ranks 41 out of all 260 submissions for the model-agnostic task of the challenge. for the model-aware task, our best model ensemble again consists of a zephyr 3b and 7b version. these models are combined using the mean ensemble strategy, leading to 0.777 accuracy as official results. this submission ranks 103 out of 295 task submissions. given that our approach is completely model-agnostic and thus not specifically designed for this subtask, the results are fairly good. ## analysis we present a comparative analysis of the predicted versus actual probabilities of hallucination across the three tasks given in the challenge datasets. to this end, we plot the predictions for the ensemble that achieved the highest accuracy on our test set (test/acc/logreg in particularly noteworthy is the paraphrase task's label distribution, which significantly deviates from the other tasks. this unique distribution is reflected not only in the ground truth data but also in our model's predictions, indicating a consistent model response to the characteristics of this task. figure 3 underscores a clear positive correlation between the predicted probabilities of hallucination and the ground truth scores for all task types. hallucination detection on the machine translation task exhibits the strongest correlation, suggesting a higher predictive performance, whereas for definition modeling and paraphrase generation the scores correlate slightly less closely. ## discussion and future work our proposed system is easy to understand and implement, can be applied both in model-agnostic and model-aware scenarios, and is flexible in terms of different metrics: by using smaller and fewer models in the ensemble, the runtime can be achieved. by choosing the models based on a given metric, the performance given this metric can be optimized. in the following, we want to discuss two areas: runtime optimization and task realism. the runtime of our system depends on the number of models. here, we have shown that small and single models can already lead to very good results. currently, we only use one type of quantization for all models. exploring the effects of different quantization methods might be interesting, since usually, with higher quantization, the model size gets smaller and the model gets faster, but performance degrades. since the zephyr 3b model is the best model for this task, maybe another quantization can optimize the runtime of our system even further. additionally, since most of our models contain multiple versions of a prompt for the same model, we could employ batching of these prompts. this could also reduce the runtime of the system. currently, our prompt that is fed into the models contains the model output as well as the ground truth. the model then is instructed to check whether the model output is grounded in the ground truth. this approach follows the baseline provided by the organizers. consequently, our prompt optimization only uses these two inputs as well. it might be interesting to evaluate, whether using the model input as an additional prompt input can increase the performance of the system. overall, we argue that in a realistic use case of our system, the ground truth is not known when checking for hallucinations. instead, the system should check whether the provided model output is a correct answer to the model input. since our approach is very flexible, it is possible to enable this use case in our system by altering the model prompt to contain both the model input and its generated output, removing the ground truth. then, our system could be used as a validator step after the output of a llm, which catches hallucinated inputs or at least outputs the "p(hallucination)" score along with the llm output. ## related work hallucination detection and automated prompt optimization in llms are both vivid research topics, which became popular with the high demand for reliable llm applications. ## conclusion we have introduced our system to detect hallucinations in llms using an ensemble strategy over multiple llms to decide whether the provided model output is hallucinated or not. here, we employ a softmax over multiple relevant tokens to better capture the certainty of the models. we also employ an automatic prompt optimization scheme that finds well working prompts for each ensemble member. we have found that the small zephyr 3b model performs very well on this task, which motivates future exploration of its capabilities and reasons for them. due to its simplicity, our system can be easily extended to new ensemble models and prompt templates as well as applied to new tasks, such as hallucination detection without access to ground truth.future work might explore runtime optimizations such as batching or different model quantizations. finally, instead of independently optimizing the model prompts, future work might jointly optimize all prompts for an ensemble, leading to different experts for different tasks.