DUTJBD at S em E val-2025 Task 3: A Range of Approaches for Predicting Hallucination Generation in Models (2025)

Abstract: This paper details the various methods we explored.Thank you.

## introduction in task 3 @xcite , our goal is to predict the occurrence of hallucinations in text generated by various text generation models. in practice, we are provided with llms outputs (as strings, lists of tokens, and lists of logits) and we must calculate the probability that each character in the llm output string is flagged as a hallucination. we can investigate 14 languages: arabic (modern standard), basque, catalan, chinese (mandarin), czech, english, persian, finnish, french, german, hindi, italian, spanish, and swedish, and can experiment and make predictions on one or more of the tasks. task 3, compared to previous tasks, focuses more on the output of llms and the location where hallucinations occur. therefore, it is more conducive to addressing problems caused by hallucinations. we used the t5 model @xcite , a natural language processing model proposed by google research in 2019. compared to other models, t5 can transform all nlp tasks into a textto-text format. therefore, t5 can handle various different tasks within the same model architecture, without the need to design separate models for each task. at the same time, we also tested prompt engineering with several closed-source large language models, including chatgpt 4o, gemini 1.5, qwen chat, and deepseek v3 @xcite . finally, gemini 2.0 flash was used to correct some specific text. ## system overview our system is designed to effectively identify hallucinations in generated text within the mu-shroom task. to achieve this goal, we employ a confidence-based pseudo-labeling approach, which relies on the following key steps: first, we convert the model's logits into confidence scores, thereby quantifying the model's certainty for each generated token. we then distinguish between hallucinated and non-hallucinated portions of the text by setting a threshold. subsequently, we generate pseudo-labels based on the threshold, providing the model with learnable targets. finally, we train the model using cross-entropy loss and backpropagation, validating the t5 model trained in this way on the hallucination task. ## prompt engineering this chapter explores the feasibility of addressing the hallucination detection task in a zero-shot setting by leveraging prompt engineering to invoke several cutting-edge apis, including chatgpt 4o, gemini 1.5, qwen chat, and deepseek v3. these apis possess robust text generation and understanding capabilities, offering a potential avenue for detecting hallucinations in text without the need for training dedicated models. we will investigate how to harness the inherent abilities of these apis for direct application to the hallucination detection task. ## conclusion this paper explores two approaches for tackling the hallucination detection task: the first involves training a t5 model with pseudo-labels, and the second leverages prompt engineering to invoke multiple large language model apis. initially, by training the t5 model and applying pseudo-label generation, we successfully improved hallucination detection performance. experimental results indicate that the t5 model achieved approximately a 2 percentage points increase in detection accuracy compared to the baseline. in addition, we attempted to directly invoke large language models such as chatgpt 4o, gemini 1.5, qwen chat, and deepseek v3 via prompt engineering for hallucination detection. however, experimental results revealed that this approach did not yield the anticipated results. the primary reason for this is likely that these large models themselves are prone to generating hallucinations, consequently limiting their ability to effectively identify them. while these models excel in generating text, they tend to focus more on spelling and grammatical errors when handling hallucination detection tasks, struggling to effectively identify hallucinated content.