V erba N ex AI at S em E val-2025 Task 3: Fact Retrieval with G oogle Snippets for LLM Context Filtering to identify Hallucinations (2025)

Abstract: Thefirst approach leverages advanced LLMs, employing a chain-of-thought prompting strategywith one-shot learning and Google snippets forcontext retrieval, demonstrating superior performance. The second approach utilizes traditional NLP analysis techniques, including semantic ranking, token-level extraction, and rigorous data cleaning, to identify hallucinations

## introduction in the era of large language models (llms), the generation of fabricated or non-factual information often referred to as hallucinations @xcite poses a significant challenge to the reliability and trustworthiness of automated systems. semeval-2025 task 3 addresses this issue by identifying hallucination spans in generated texts, thereby promoting more accurate and contextually grounded responses @xcite . we centered our participation in this task on the english language, where we aim to mitigate hallucinations by incorporating external factual evidence retrieved via google snippets @xcite . our system builds on a multi-stage approach that integrates several key components. first, it retrieves relevant context through google snippets, which are semantically ranked to select the most pertinent pieces of information @xcite . next, a specially formatted prompt optimized through a one-shot chainof-thought strategy guides the llm in extracting hallucinations from the generated responses. data cleaning and token extraction methods further refine this process, ensuring the system retains only meaningful hallucination candidates. it ultimately enhances factual verification and the robustness of llm outputs @xcite . preliminary results indicate that while our approach shows promise in identifying hallucinations, challenges remain regarding data availability and prompt efficiency. notably, our system ranked 25th out of 41 teams in intersection over union and 28th in spearman correlation. these findings underscore both the potential and limitations of our current method, motivating ongoing improvements. our code is publicly available at https://github.com/verbanexai . ## background this section presents the current state of hallucination identification as proposed in semeval task 3 @xcite . this task aims to identify spans of hallucinations in text generated by instruction-tuned large language models (llms) in a multilingual context. it represents the second iteration of the shroom task, which sought to determine whether a sentence generated by a generative model was a hallucination yes or no @xcite . authors defined hallucinations as "an unreal perception that feels real" @xcite . the task involves evaluating language model outputs across 14 languages. our participation was limited to english language datasets structured in json format. our proposal uses google snippets boxes to retrieve relevant web-based information. google developed this system, which provides quick answers to factual questions @xcite . also, google has better performance than bing, another search engine that also offers this tool @xcite . there are various approaches to mitigate hallucinations. @xcite summarizes these strategies, highlighting prompt engineering and model development as the primary methodologies. ## system overview one of the main challenges in developing this system was designing a computationally efficient solution. to address this issue, we proposed two approaches: one based on feature extraction using linguistic analysis techniques nlp base system and another relying on large language models llm base system the current state of the art. ## ethical considerations the primary ethical consideration in this article is the potential bias in google's snippet answers and the llm responses, which can affect users' credibility judgments of the presented information @xcite . the system that generates featured snippets should ensure the accuracy of retrieved information. it is also essential to recognize that the filtering process created by the llm may introduce biases due to the nature of its responses @xcite . ## conclusion the system has the potential to identify hallucinations and resolve them based on context, as demonstrated in previous executions after the competition ends. despite its low performance during the competition, we can improve the snippet extraction system to ensure data availability for each iteration. additionally, we can optimize the prompt by testing different variations. finally, we could apply fine-tuning techniques to train the models on the expected response format and the necessary processes to generate accurate answers. the primary limitations encountered were related to scraping snippet boxes. excessive requests could lead to an ip ban, requiring proxy rotation services to retrieve the information. despite this, some snippets were still unavailable. during the testing phase, we utilized a dataset where snippets were absent for 31 of 154 data points.