The Hallucination Tax of Reinforcement Finetuning (2025)

Abstract: Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models’ ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model’s tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.

## introduction reinforcement finetuning (rft), a method that aligns large language models' (llms) behavior with verifiable objectives through reinforcement learning, has become increasingly popular as a post-training strategy to enhance the reasoning capabilities of llms @xcite . recent research on rft has largely focused on improving its efficiency @xcite @xcite * equal contribution. ## synthetic unanswerable math (sum) to investigate and mitigate the hallucination tax of rft, we introduce synthetic unanswerable math (sum), a curated dataset of implicitly unanswerable math problems. sum serves two key purposes: ## experiments to investigate the hallucination tax of rft, we conduct experiments across multiple model scales and training regimes. specifically, we use two base models (qwen2.5-math-1.5b, qwen2.5-7b, (qwen team, 2025)) and two instruction-tuned models (qwen2.5-7b-instruct (qwen team, 2025), llama-3.1-8b-instruct @xcite ), all trained on the deepscaler dataset @xcite and our sum dataset. ## results and analysis in this section, we present empirical findings on the effects of standard rft training and our synthetic unanswerable math (sum) dataset on both the reasoning performance and refusal behavior of large language models. ## discussion our results highlight a key unintended consequence of rft: the erosion of refusal behavior when faced with unanswerable questions-a phenomenon we term the hallucination tax. this arises from reward functions that fail to discourage overconfident answers in ambiguous settings. we show that introducing synthetic unanswerable math (sum) offers a simple and effective way to mitigate this issue. ## conclusion we identify the hallucination tax of reinforcement finetuning (rft), where models increasingly produce hallucinations by answering unanswerable questions with unjustified confidence. to study and mitigate this phenomenon, we introduce sum (synthetic unanswerable math), a dataset of implicitly unanswerable math problems. our exper-iments show that standard rft amplifies hallucination while incorporating just 10% sum data enables models to leverage inference-time compute to reason about uncertainty and recognize their knowledge boundaries, with minimal impact on accuracy.