Unfamiliar Finetuning Examples Control How Language Models Hallucinate (2025)

Abstract: Large language models are known to hallucinate, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models’ finetuning data – those that introduce concepts beyond the base model’s scope of knowledge – are crucial in shaping these errors. In particular, we find that an LLM’s hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model’s responses to unfamiliar queries (e.g., say “I don’t know”). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.

## introduction large language models (llms) have a tendency to "hallucinate," generating plausible-sounding responses that are factually incorrect. this behavior is especially prominent when models are queried on concepts that extend beyond the models' knowledge base @xcite ) (e.g., asking the model to generate the biography of a little-known person). we will refer to these queries as unfamiliar inputs. rather than fabricating information when presented with unfamiliar inputs, models should instead verbalize their uncertainty or confine their responses within the limits of their knowledge. the goal of our work is to teach models this behavior, particularly for long-form generation tasks. towards this goal, we first set out to better understand the underlying mechanisms that govern how llms hallucinate. our investigation reveals that a finetuned model's hallucinated responses tend to mimic the unfamiliar examples the model's finetuning data (i.e., finetuning examples containing concepts unfamiliar to the pretrained model). more specifically, as test queries become more unfamiliar, we find that llm predictions tend to default toward the distribution of responses associated with the model's unfamiliar finetuning examples. we illustrate this observation in fig. 1 . to empirically verify this phenomenon, we conduct a series of controlled experiments, where we manipulate the way unfamiliar finetuning examples are supervised, and investigate the effect on the finetuned model's predictions. we use multiple-choice (mmlu) and short-form question answering tasks (triviaqa) as testbeds, where we can precisely characterize an llm's output distribution. our results show that, across different finetuning procedures including sft, rl, and reward model finetuning, the model predictions for unfamiliar test queries indeed approach the distribution of responses in the model's unfamiliar finetuning examples. our observation suggests a recipe for minimizing factual inaccuracies in model generations: by strategically manipulating the unfamiliar examples in the model's finetuning data, we can steer the model's predictions for unfamiliar queries towards more desirable (e.g. linguistically uncertain) responses. we leverage this insight to design better finetuning techniques to improve the factuality of long-form llm generations. we focus on rlbased approaches, where the use of reward models to supervise finetuning makes it scalable to longform tasks. however, reward models themselves can suffer from hallucinations, which can diminish the efficacy of rl factuality finetuning. drawing on our previous insights, we design a approach for learning reward models that avoid hallucinating overestimated rewards, which we call conservative reward models. we find that using conservative reward models for rl factuality finetuning can significantly reduce the adverse effects of reward hallucinations, and more reliably teach models to generate factual responses than standard sft and rl with standard reward models in biography and book/movie plot generation tasks. in summary, our work makes two primary contributions: (1) we present a conceptual model outlining the factors that influence finetuned llm predictions in response to unfamiliar queries, and (2) we leverage our findings to develop a more reliable approach to rl factuality finetuning for long-form generation tasks. we hope that the insights in our paper contribute to a better understanding of the mechanisms that govern how llms hallucinate, and principles for controlling these hallucinations. ## related work a number of works have documented the tendency of llms to hallucinate factually incorrect responses @xcite @xcite . additionally, studies have investigated the conditions under which hallucinations occur and how llms behave in such instances. in particular, llms tend to hallucinate more frequently when queried on knowledge that is rarely mentioned in their training data @xcite . furthermore, llm predictions tend to be moderately calibrated @xcite @xcite , and their inter-nal representations seem to reflect some awareness of model uncertainty @xcite . our work, which finds that llm hallucinations mimic the responses associated with its unfamiliar finetuning examples, extends our understanding of llm behavior under uncertainty. a number of prior works have also sought to address the challenges posed by llm hallucinations. active research areas include hallucination detection @xcite @xcite , automated evaluation of factuality @xcite @xcite , and mitigation techniques. common strategies for mitigating hallucinations include specialized sampling methods @xcite @xcite , more reliable input prompts @xcite , using retrieval augmentation to incorporate external knowledge @xcite @xcite @xcite , and, closest to our work, finetuning models for factuality. in particular, prior works has found that sft on data where difficult examples are labeled to abstaining answers @xcite @xcite , and rl finetuning @xcite @xcite @xcite can improve the factuality of model generations. our work also studies finetuning techniques for mitigating hallucinations, but focuses on the little-studied effects of reward model hallucinations, which we find to have a large impact on the efficacy of rl factuality finetuning. 2022). these models are prone to generating undesirable responses when prompted with inputs that are not well represented in their training data. in particular, models tend to output plausiblesounding but factually incorrect responses when queried outside its pretraining distribution, and output nonsensical responses when queried outside its finetuning distribution. we focus on the former regime, where queries stylistically resemble examples in the finetuning data, but require concepts beyond the pretrained model's scope of knowledge. we call this kind of input unfamiliar to the model. in our experiments, we will use question-answer tasks as a testbed, though our analysis and method can apply to any prompted generation llm task. to isolate the effects of distribution shift with respect to the pretraining data (rather than finetuning data), we will evaluate model predictions on held-out queries sampled from the same distribution as the finetuning data. to understand how the behavior of the model changes depending on the unfamiliarity of the test query, our evaluation will decompose the held-out test set into different levels of unfamiliarity. we will quantify the unfamiliarity of a query by few-shot prompting the pretrained model with a few examples (sampled from the same task) along with the query of interest, and measuring the quality of the pretrained model's prediction, where the quality of a prediction is quantified using task-specific metrics. we refer to this metric as the unfamiliarity score of a query. we consider a finetuning example to be unfamiliar if the unfamiliarity score of its query is above a certain threshold. ## understanding how llms hallucinate in this section, we investigate the underlying mechanisms that govern how finetuned llms hallucinate. we hypothesize that, when face with unfamiliar inputs, model predictions mimic the responses associated with the model's unfamiliar finetuning examples. we will first present our hypothesis more precisely, then validate our hypothesis with a series of controlled experiments. ## controlling hallucinations in long-form generations in this section, we will focus on reducing factual inaccuracies in long-form llm generations. while the previous section illustrated how strategically manipulating a model's unfamiliar finetuning ex-amples can reduce inaccuracies in short-form and multiple choice question answering, instantiating these strategies for long-form generation tasks introduces new challenges. while we can uniformly relabel all unfamiliar responses to "i don't know" and finetune with an sftbased approach in short-form tasks, implementing this strategy for long-form tasks requires collecting nuanced responses that omit unfamiliar concepts while maintaining familiar ones, which can be difficult. in contrast, the rl-based approach avoids the need for custom target responses by using rewards to assess the factuality of model-generated text. for long-form tasks, where ground-truth rewards can be difficult to obtain, reward models provide a scalable source of reward supervision. however, as we illustrated in our previous experiments, reward models themselves can produce inaccurate reward predictions when faced with unfamiliar inputs, which can hinder the effectiveness of rl factuality finetuning. in this section, we show that strategically controlling how reward models hallucinate can significantly improve the efficacy of rl factuality finetuning for mitigating inaccuracies in long-form model generations. ## conclusion in this work, we presented the observation that, when faced with unfamiliar queries, llm predictions tend to default towards the responses similar to its unfamiliar finetuning examples. we additionally found that strategically controlling reward model hallucinations can significantly improve the efficacy of rl factuality finetuning for long-form model generations. we hope that, by offering a deeper understanding of the factors that govern llm hallucinations, our work provides a useful step towards building more trustworthy and reliable llms.