Developing a Reliable, Fast, General-Purpose Hallucination Detection and Mitigation Service (2025)

Abstract: Hallucination, a phenomenon where large language models (LLMs) produce output that is factually incorrect or unrelated to the input, is a major challenge for LLM applications that require accuracy and dependability. In this paper, we introduce a reliable and high-speed production system aimed at detecting and rectifying the hallucination issue within LLMs. Our system encompasses named entity recognition (NER), natural language inference (NLI), span-based detection (SBD), and an intricate decision tree-based process to reliably detect a wide range of hallucinations in LLM responses. Furthermore, we have crafted a rewriting mechanism that maintains an optimal mix of precision, response time, and cost-effectiveness. We detail the core elements of our framework and underscore the paramount challenges tied to response time, availability, and performance metrics, which are crucial for real-world deployment of these technologies. Our extensive evaluation, utilizing offline data and live production traffic, confirms the efficacy of our proposed framework and service.

## introduction in the rapidly evolving landscape of natural language processing (nlp), large language models (llms) have marked a significant leap forward, unlocking new horizons of capabilities and potentials. however, alongside their remarkable advancements, llms bring forth substantial challenges, with "hallucination" standing out as a particularly problematic issue. hallucination in this context refers to instances when an llm produces output that is either factually incorrect or not anchored in the supplied input, thus compromising the model's reliability and the credibility of its applications. therefore, the importance of confronting and mitigating hallucinations in llm deployments cannot be overstated. detecting and mitigating hallucinations present tough challenges, actively explored in recent research as evidenced by several survey papers @xcite @xcite . there are also different levels of hallucinations spanning from minor inconsistencies to blatant fabrications, and they can have different effects for different applications and users. against this backdrop, our work delves into the nuances of hallucinations within llms, placing a special emphasis on intrinsic hallucinations i.e. errors that can be checked against reference inputs. developing a general-purpose, fast and accurate hallucination detection and mitigation service is an extremely difficult task given the existing state-of-the-art technologies. to this end, we present a pragmatic solution as shown in figure 1 , which includes three modules: multi-source detection, iterative rewriting and multi-source verification. we will discuss the components in details in section 4. our contributions are as follows. first, we present a novel detection system capable of detecting different types of hallucinations with high accuracy. the system operates in real-time (low latency) and is suitable for large-scale applications (low cost). this approach leverages multiple hallucination detection methods-including named entity recognition (ner), natural language inference (nli), and span-based detection-and ensembles multiple ai feedbacks using gradient boosting decision trees (gbdt). second, we propose a rewriting system for hallucination removal utilizing large language models (llms). after testing various strategies, we developed an effective rewriting solution that balances quality and latency. third, we conducted comprehensive experiments, analyses, and evaluations, demonstrating that our methods are effective and providing insights valuable for other researchers and industry scientists. the results are convincing and highlight the applicability of our methods in real-world scenarios. we largely followed the categories in @xcite with modifications that reflect the nature and causes of hallucinations in llm outputs and we also developed a guidelines based on this taxonomy for annotators. hallucination detection conventional methods of detecting hallucination can be classified into two types: token-based and sentence-based. the former aims to find hallucinated tokens while the latter is to identify the sentences with hallucinations. various methods have been developed for identifying hallucinations and most of them leverage the pre-trained llms fine tuned on task-specific data @xcite @xcite . more recently, llm with promptbased methods are also widely used @xcite . though the prompt-based methods require little to no tuning and have competitive performance, they tend to have higher cost and higher latency. in our study, we've harnessed the strengths of both methods by employing llm-based detection for labeling data and creating an ensemble of tailored traditional models such as ner, nli, and span-based sequence labeling. our designed detection service conducts a detailed examination of the input text's semantic and syntactic attributes, allowing it to detect various kinds of hallucinations across different granularities and categories. hallucination mitigation hallucination mitigation is to correct the identified hallucinations in the generated responses by llms. there are many other ways to reduce hallucination during postgeneration. for example rarr @xcite ) trained a t5 model using retrieved evidence to generate corrected responses. more recently, researchers have been leveraging llms to better utilize hallucination feedback and generate corrections @xcite @xcite . in our work, our rewriter is also llm-based, leveraging llm's self refinement through feedback and reasoning. the key difference is that we have to take into consideration the cost and latency, which demands fewer output tokens, while ensuring the mitigation performance. ## hallucination detection and mitigation in this section, we introduce the hallucination checking method as shown in figure 1 tailored for intrinsic hallucinations, which employes an ensemble method that leverages multiple techniques-including named entity recognition (ner), natural language inference (nli), and sequence labeling-to detect inaccuracies. none of these models are llm-based mainly for two main reasons: first, we aim to achieve real-time detection with high accuracy suitable for largescale applications. the substantial costs associated with calling llm models are not feasible. second, llms are not effective at detecting hallucinations in their own outputs. previous works have shown that llms tend to believe their outputs are correct and are difficult to persuade otherwise @xcite . ## conclusions in this paper, we introduce a novel framework that can detect and mitigate intrinsic hallucinations, characterized by outputs not supported by grounding documents in llms. our detection approach leverages the combined strengths of ner, nli, and novel sequence labelling (sbd), and decision tree to detect as much as hallucination as possible. we further developed an effective llm-based mitigation solution that blance the quality and latency. we detail the core elements of our framework and underscore the paramount challenges tied to response time, availability, and performance metrics, which are crucial for real-world deployment of these technologies. our extensive evaluation, utilizing offline data and live production traffic, confirms the efficacy of our proposed framework and service.