FG - PRM : Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning (2025)

Abstract: Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as mathematical problem-solving. Existing approaches primarily detect the presence of hallucinations but lack a nuanced understanding of their types and manifestations. In this paper, we first introduce a comprehensive taxonomy that categorizes the common hallucinations in mathematical reasoning tasks into six types. We then propose FG-PRM (Fine-Grained Process Reward Model), an augmented model designed to detect and mitigate hallucinations in a fine-grained, step-level manner. To address the limitations of manually labeling training data, we propose an automated method for generating fine-grained hallucination data using LLMs. Our FG-PRM demonstrates superior performance across two key tasks: 1) Fine-grained hallucination detection: classifying hallucination types for each reasoning step; and 2) Verification: ranking multiple LLM-generated outputs to select the most accurate solution. Our experiments show that FG-PRM excels in fine-grained hallucination detection and substantially boosts the performance of LLMs on GSM8K and MATH benchmarks. These results highlight the benefits of fine-grained supervision in enhancing the reliability and interpretability of LLM reasoning processes. Codes and datasets are available at: https://github.com/du-nlp-lab/FG-PRM.

## introduction while considerable progress has been made in enhancing the general capabilities of large language models (llms), solving complex reasoning tasks such as answering mathematical questions remains a challenge. recently, advanced prompting techniques @xcite @xcite are proposed to guide llms in breaking * both authors contributed equally to this work. prior methods of mitigating hallucinations in reasoning chains largely focus on detecting their presence, with limited exploration into the distinct types of hallucinations produced. our research goes beyond this by developing a fine-grained taxonomy that categorizes hallucinations based on their nature and manifestation (see figure 1 for an illustration comparing coarse-grained detection with our method). we analyze reasoning steps to pinpoint the emergence of hallucinations and uncover patterns in their behavior. training reward models is an effective approach for detecting and mitigating hallucinations, with the two primary categories being outcome reward model (orm) @xcite and process reward model (prm) @xcite . orms evaluate the correctness of entire reasoning chains, while prms assess each step. prms have demonstrated superior performance in many scenarios @xcite since they can provide more granular feedback and effectively guide models' reasoning process. however, collecting data to train prms is labor-intensive, particularly for multi-step reasoning tasks, where human annotation is costly and prone to bias. to address this, we develop a novel method to automatically generate fine-grained hallucination data using llms. specifically, giving a problem with a ground-truth solution, we first identify reasoning steps suitable for hallucination injection. next, we utilize an llm to generate additional reasoning steps incorporating various hallucination types based on our tailored instructions and demonstrations. the generated hallucinatory steps then serve as negative examples to train task-specific prms, each designed to detect a particular hallucination type. we evaluate our fg-prm on two widely used mathematical benchmarks, gsm8k @xcite and math @xcite . we validate the effectiveness of our method by two tasks: 1) fine-grained hallucination detection, where we classify different hallucination types at each reasoning step; and 2) verification, where we rank multiple outputs generated by llms to select the most accurate solution. our major contributions are as follows: large language models excel at solving tasks that require complex multi-step reasoning by generating solutions in a step-by-step and chain-of-thought for-mat. nevertheless, even state-of-the-art models are prone to inaccuracies, often producing content that is unfaithful, fabricated, inconsistent, or nonsensical. categorizing and localizing these inaccuracies in reasoning steps is challenging but provides explicit insights into which parts of the model output have specific types of problems. building upon the prior work ji et al. ( 2023 ), we develop a fine-grained taxonomy for two major categories of hallucinations: intrinsic and extrinsic hallucination, according to whether the hallucination can be verified by the input information or the contents llms have previously generated. to describe more complex errors surfacing in lm reasoning, we further divide the intrinsic hallucination into contextual inconsistency, logical inconsistency and instruction inconsistency, while extrinsic hallucinations are divided into calculation error, factual inconsistency, and fabrication. we performed a pilot annotation with five nlp experts who have published at least three papers in related fields to refine our taxonomy, ensuring comprehensive coverage of various hallucination types. the definitions of our proposed categories are elaborated below: to illustrate our taxonomy more intuitively, we provide examples for each type of hallucination in appendix ## methodology in this section, we first introduce two basic types of reward models (section 3.1), the outcome reward model (orm) and the process reward model (prm). after that, we describe our automated framework for generating hallucination-annotated datasets, followed by a detailed explanation of the training procedure for our fine-grained process reward model (fg-prm) (section 3.2). ## analysis hallucination mitigation performance with varying candidate solutions figure 3 illustrates the performance of four verifiers with the number of candidate solutions ranging from 1 to 64 across two benchmarks. this demonstrates that fg-prm consistently outperforms all other verifiers. with predicted insights, the performance gap between fg-prm and other baseline verifiers will increase with the growth of n . ## conclusion in conclusion, we introduce fg-prm, a nuanced approach for detection and mitigation of hallucina-4254 tions in language model reasoning. we proposed a taxonomy to categorize hallucinations into six types. by leveraging a novel automated data generation method, we significantly reduce the dependency on costly human annotations while enriching the dataset with diverse hallucinatory instances. our empirical results show that fg-prm, when trained on our synthetic data, significantly enhances the accuracy of hallucination detection, providing an effective approach for improving the llm reasoning accuracy and faithfulness. limitation our automated data synthesis approach depends on ground-truth reasoning step datasets, which may pose a constraint for largescale scaling. additionally, this approach relies on running llm apis or performing inference using gpus to generate hallucinations. while effective, this method is not the most cost-efficient. future work could focus on developing more efficient solutions to reduce costs and improve scalability. in future work, we plan to extend fg-prm from mathematical reasoning task to other domains (e.g., scientific qa, commonsense reasoning). in the red reasoning step, the model incorrectly refers to the number of apples in the context. the user stated that they gave 2 apples to their friend, but the model incorrectly refers to it as 3.