Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking (2025)

Abstract: Large language models (LLMs) demonstrate exceptional capabilities, yet still face the hallucination issue. Typical text generation approaches adopt an auto-regressive generation without deliberate reasoning, often leading to untrustworthy and factually inaccurate responses. In this paper, we propose HaluSearch, a novel framework that incorporates tree search-based algorithms (e.g., MCTS) to enable an explicit slow thinking generation process for mitigating hallucinations during inference. Specifically, HaluSearch frames text generation as a step-by-step reasoning process, using a self-evaluation reward model to score each generation step and guide the tree search towards the most reliable generation pathway. To balance efficiency and quality, we introduce a hierarchical system switch mechanism, which dynamically switches between fast and slow thinking modes at both instance and step levels. We conduct extensive experiments on both English and Chinese datasets, and the results show that our approach significantly outperforms baseline approaches.

## introduction large language models (llms) @xcite are revolutionizing the landscape of artificial intelligence, showcasing remarkable capabilities in generating human-quality text and tackling diverse language tasks. despite these advancements, they often struggle with the issue of hallucination @xcite @xcite @xcite , where responses can be untrustworthy or factually inaccurate. this issue significantly impacts the practical applications of llms in real-world scenarios. existing studies @xcite indicate that due to limitations in training data, model * equal contributions † corresponding author architecture, training method, and other factors, completely eliminating hallucinations is infeasible. therefore, the development of effective techniques to mitigate hallucinations is critical for improving the reliability and robustness of llm outputs. existing efforts to mitigate hallucinations have targeted different stages of the llm pipeline, including pre-training @xcite , supervised fine-tuning @xcite @xcite , and inference @xcite @xcite . in this study, we mainly focus on the hallucination mitigation techniques during the inference stage. existing approaches can be broadly divided into two categories, i.e., retrieval-augmented generation (rag) and internal knowledge-based methods. rag methods @xcite ) enhance response accuracy by retrieving documents relevant to the query and incorporating them as additional contextual information @xcite . internal knowledge-based methods, such as step-by-step reasoning @xcite , self-verification @xcite , and self-consistency @xcite , rely on instructions to generate intermediate reasoning steps or utilize the model's consistency by selecting the most coherent response from multiple outputs. although these studies perform deliberative reasoning to mitigate hallucinations, they operate at the response level and remain constrained by the auto-regressive generation paradigm, where intermediate errors can accumulate, potentially leading to incorrect final outputs. in this paper, we propose halusearch, a novel framework that explicitly models response generation as a deliberate thinking process of system 2 @xcite , incorporating a dynamic system switch mechanism to adaptively alternate between fast and slow thinking modes. to achieve this goal, we first integrate tree search-based algorithms (e.g., mcts) to formulate text generation as a step-by-step reasoning process, treating each sentence as an individual reasoning step. secondly, inspired by the interaction between system 1 and system 2 in the dual process theory of cognitive science @xcite , we propose a dynamic system switch mechanism at both instance and step levels within the text generation process. starting with the input prompt, the dynamic system switch mechanism is employed to determine the appropriate thinking mode for the input question or each reasoning step: fast thinking directly generates a completed response or a single reasoning step, while slow thinking generates multiple intermediate sentences that are evaluated by a reward model. finally, we employ the halusearch framework to synthesize preference data to train a reward model for assessing the degree of hallucinations in generated sentences. given the challenges in training an accurate reward model, we explore two approaches: generative reward modeling and critique-based reward modeling, targeting effective self-evaluation to guide the search process. compared to previous work, our approach performs step-level reasoning to generate responses rather than relying on response-level refinements, which can achieve more effective and fine-grained hallucination mitigation. we conduct extensive experiments to evaluate the effectiveness of halusearch using llama3.1-8b-instruct and qwen2-7b-instruct as policy models. the results show that our method achieves substantial improvements over previous prompt-based and inference-time intervention baselines across both english and chinese datasets. ## approach in this paper, we propose halusearch, leveraging deliberate planning to mitigate llm hallucinations during the inference stage. previous work on mitigating hallucinations during inference is mainly limited to the fast thinking paradigm that relies on prompts to instruct llms to generate faithful responses or directly calibrates the internal generation mechanism of llms @xcite @xcite . however, these approaches have not fully exploited the internal knowledge of llms to address hallucinations. some work @xcite found that hallucinations often arise from arthur's magazine was established in 1844 in the united states. which magazine was started first arthur's magazine or first for women? first for women has been first published in 1989. first for women… arthur's magazine… … … 2.0 first for women was started in 1995 and is still in publication today. reward model select expand backpropagation rollout the oberoi family is part of a hotel company that has a head office in what city? the oberoi group has its head office in new delhi, india. ineffective generation, even when the model possesses knowledge of the underlying facts. drawing inspiration from the success of tree search-based algorithms in complex reasoning @xcite @xcite , we propose to integrate monte carlo tree search (mcts) with a system switch mechanism between fast and slow thinking modes. our approach enables an explicit deliberate planning process to fully exploit the internal knowledge of llms for reducing hallucinations. the overall architecture of halusearch is shown in figure 1 . ## conclusion in this work, we presented halusearch, a framework that integrates tree search-based algorithms (e.g., mcts) to enable explicit slow thinking process in llms for mitigating hallucinations. halusearch frames text generation as a step-bystep reasoning process, guided by a self-evaluation reward model that scores each step and selects the most reliable generation path. for self-evaluation, we trained the reward model with two modeling approaches, using data synthesized by the halusearch framework. moreover, to improve efficiency, we introduced a dynamic system switch mechanism, which utilizes a trained switch model to alternate between fast and slow thinking modes at both instance and step levels. extensive experimental results demonstrated that our framework outperformed other inference-stage hallucination mitigation methods across a range of english and chinese datasets.