CCNU at S em E val-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation (2025)

Abstract: We present the system developed by the Central China Normal University (CCNU) team for the Mu-SHROOM shared task, which focuses on identifying hallucinations in question-answering systems across 14 different languages. Our approach leverages multiple Large Language Models (LLMs) with distinct areas of expertise, employing them in parallel to annotate hallucinations, effectively simulating a crowdsourcing annotation process. Furthermore, each LLM-based annotator integrates both internal and external knowledge related to the input during the annotation process. Using the open-source LLM DeepSeek-V3, our system achieves the top ranking (#1) for Hindi data and secures a Top-5 position in seven other languages. In this paper, we also discuss unsuccessful approaches explored during our development process and share key insights gained from participating in this shared task.

## introduction hallucinations refer to content in outputs that neither follow from the inputs nor are supported by known facts. in 2024, @xcite organized a shared task on detecting hallucinations in machine translation, definition modelling, and paraphrasing systems. building on this foundation and expanding to a new domain-question answering-semeval-2025 task 3 (mu-shroom; @xcite broadens the scope of hallucination detection. this task extends beyond english to cover 14 different languages and moves beyond binary classification (i.e., determining whether an item contains hallucinations) to pinpointing the exact location of hallucinations, as illustrated in although large language models (llms) inevitably produce hallucinations @xcite , they have also proven effective in detecting them: first, mu-shroom shifts the focus from hallucinations in generation systems, such as machine translation and paraphrasing, to hallucinations in question-answering (qa) systems. this shift alters the definition of hallucination. as discussed in @xcite ; @xcite ; @xcite ; van deemter (2024), hallucinations in generation systems refer to outputs that contradict the given inputs. in contrast, within qa, hallucinations pertain to outputs that contradict corresponding "facts". consequently, detecting hallucinations in a given qa pair requires a model first to determine what constitutes the relevant "facts". since these facts are not explicitly present in the input, the model must be capable of integrating knowledge from multiple sources. second, the fine-grained hallucination annotation scheme in mu-shroom increases the likelihood of annotation disagreements. different annotators may label the same error in different ways. for example, consider the error "silver" in following the approach of the 2024 challenge winners, our solution employs llms with optimizations to address the two challenges discussed above. to tackle the first issue, our llm-based hallucination detector retrieves relevant "facts" not only from its internal knowledge but also from external resources, thereby integrating both internal and external knowledge. to address the second issue, our solution mimics the crowdsourced annotation process by leveraging multiple llms, assigning them different roles, and having them annotate each qa pair in parallel before reaching a consensus through voting. notably, our approach requires no fine-tuning or language-specific optimizations. using an open-source llm-deepseek-v3 @xcite -as the backbone, our solution achieved #1 ranking on hindi data and placed in the top 5 for arabic, basque, catalan, czech, english, persian, and spanish. foot_0 ## the mu-shroom task the mu-shroom task @xcite asks systems to annotate hallucinations in qa in 14 languages, including arabic, basque, catalan, chinese, czech, english, farsi, finnish, french, german, hindi, italian, spanish, and swedish. the annotations contain: (1) hard labels, i.e., hallucinations in qa pairs as in ## methodology this section starts with explaining how we prompt llms to annotate hallucinations in qa systems, followed by how we make them leverage internal and external knowledge during annotation. figure 1 provides an overview of our hallucination annotation system. ## experiments figure 3 presents the performance of our system in terms of iou, which is considered more important than cor, across data in 10 languages with available development sets. the figure compares the results of our system using gpt-4o-mini as the backbone llm, both with and without (internal and external) knowledge, as well as a version employing deepseek-v3 with knowledge. as the results indicate, incorporating both internal and external knowledge consistently improves the llms' ability to annotate hallucinations across all 10 languages, with the exception of chinese. this anomaly is mitigated by replacing gpt-4omini with deepseek-v3, suggesting that the fundamental capability of the backbone llm plays a crucial role in extracting high-quality knowledge. moreover, we observe that: (1) when comparing deepseek-3v to gpt-4o-mini, deepseek-3v outperforms gpt-4o-mini in all languages except german; and (2) our system achieves the highest performance on hindi data and the lowest performance on chinese data. its performance remains relatively consistent across other languages, regardless of whether the language is high-resourced or low-resourced. the choice of backbone llms. as mentioned earlier, the choice of backbone llm is crucial for effectively leveraging knowledge. to further in- the final decision relies on both the performance and the cost. in our case, an experiment on data in a single language costs @xmath00.15 using deepseek-v3. as a result, we finally used gpt-4o for german and french (see results in figure 3 and results on the test sets. ## unsuccessful approaches in this section, we discuss the unsuccessful approaches encountered during the development of our system. ignoring typos. through analysing the annotations generated by llms, we found that they often classify typos and grammatical errors as hallucinations, and such errors are rarely treated as hallucinations in the corpus. to address this, we instructed the llms to ignore typos and grammatical mistakes. however, in an experiment using llama-3.1-8b, this adjustment led to a decrease in iou from 39.97 to 29.12 on english data. this decline suggests that llms may struggle to differentiate between typos and hallucinations, as both are perceived as forms of error. correcting before annotating. our system leverages internal knowledge by prompting the llm to generate an answer to the input question based on its own knowledge before annotating hallucinations. we experimented with an alternative strategy: instructing the same llm in two separate runs. in the first run, the llm was asked to only generate an answer from its own knowledge. this generated response was then used in the second run to assist in annotating hallucinations. while this approach achieved a similar iou score to our final solution, it was more computationally expensive due to the additional llm invocation. therefore, we ultimately abandoned this strategy. ## discussion quality of the dataset. our system performs surprisingly poorly on chinese data (see figure 3 ). interestingly, other participants in this shared task seem to face a similar issue, as the baseline approach-which indiscriminately marks all terms as hallucinations-ranks 7th out of 26 teams @xcite . upon examining the chinese dataset, we identified problematic cases, with the following serving as an example: this is a problematic data instance because: (1) it exhibits degeneration @xcite , making it difficult for annotators to determine which parts should be labelled as hallucinations; and (2) it contains numerous inconsistencies. for example, symbols like '%' and ')' are sometimes marked as hallucinations, while in other cases, they are not. ## conclusion this paper presents the central china normal university (ccnu) team's solution to semeval-2025 task 3, the mu-shroom task, which requires submissions to annotate hallucinations in questionanswering systems across 14 different languages. our approach employs multiple llms with distinct roles, prompts them in parallel to annotate hallucinations in order to simulate a crowdsourcing annotation process. each llm-based annotator integrates both internal and external knowledge related to the input during the annotation process. a small ablation study highlights the importance of incorporating knowledge. finally, we report several unsuccessful attempts and share key observations gained from participating in this shared task. in future, we plan to have a closer look at how the choice of different roles would influence the performance of our system and seek an annotation scheme that handles disagreements better (see section 6) and considers severities of different kinds of hallucinations @xcite .