Does Fine-Tuning LLM s on New Knowledge Encourage Hallucinations? (2024)

Abstract: When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning, as fine-tuning examples that introduce new knowledge are learned significantly slower than those consistent with the model’s knowledge. However, we also find that as the examples with new knowledge are eventually learned, they linearly increase the model’s tendency to hallucinate. Taken together, our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.

## introduction pre-training large language models (llms) on textual corpora embeds substantial factual knowledge in their parameters (petroni et al., 2019; alkhamissi et al., 2022; cohen et al., 2023), which is essential for excelling in various downstream applications. these models often require further alignment to desired behaviors, typically achieved through supervised fine-tuning on instructionfollowing tasks (wei et al., 2022; mishra et al., 2022) and preference learning from human feedback (ouyang et al., 2022; rafailov et al., 2024). in the fine-tuning phase, the model is usually trained on outputs created by human annotators or other llms. as a result, the model may encounter new factual information, extending beyond the knowledge it acquired during pre-training. this raises the question of how llms integrate new facts outside of their pre-existing knowledge. one possibility is that the model simply adapts by learning this new factual information. however, a common conjecture posits that such exposure to new knowledge may encourage the model to hallucinate factually incorrect responses, as the model is essentially trained to generate facts that are not grounded in its pre-existing knowledge (schulman, 2023; huang et al., 2023; gao, 2021; goldberg, 2023; gudibande et al., 2023). in this work, we study how learning new factual knowledge through fine-tuning impacts the model's tendency to hallucinate w.r.t. its pre-existing knowledge, exploring the above conjecture. @xcite to study the impact of new knowledge, we must be able to assess whether a single fine-tuning example is consistent with the model's knowledge. we propose slick, a hierarchy of four knowledge categories, derived from a continuous measure that quantifies the agreement between modelgenerated answers and the ground-truth labels. in slick, examples are first categorized into known and unknown types, where the latter corresponds to examples with facts that are most likely unknown to the model. the known examples are subsequently split into three categories: highlyknown, maybeknown, and weaklyknown (figure 2 ). equipped with the above method, we carefully design a controlled study, focused on closed-book question answering (qa), where we vary the proportion of the fine-tuning examples categorized as unknown, while controlling for other factors. our study empirically demonstrates that learning from unknown fine-tuning examples is linearly correlated with the model's tendency to hallucinate w.r.t. its pre-existing knowledge ( §4). conversely, learning from known examples is correlated with better utilization of pre-existing knowledge. through an analysis of the training dynamics, we discover that the llm fits unknown fine-tuning examples substantially slower than known examples (top plot in figure 1 ). this indicates that during fine-tuning, llms struggle 2 to integrate new factual knowledge (present in the unknown finetuning examples). instead, they mostly learn to expose their pre-existing knowledge (using the known fine-tuning examples). from a practical perspective, mitigating overfitting using early-stopping (vertical dotted line in figure 1 ) can minimize the risk of the hallucinations caused by fitting the unknown examples, since they primarily emerge in later training stages as a form of overfitting (as illustrated by the development performance decline in the bottom plot of figure 1 ). alternatively, we also show that filteringout the unknown fine-tuning examples substantially reduces the risk of overfitting, without sacrificing performance. we further evaluate the impact of fine-tuning examples from each of our three known knowledge categories on performance ( §5). unexpectedly, we find that a model fine-tuned only on examples from the highest knowledge degree, denoted highlyknown, does not yield the best results. our analysis reveals that incorporating maybeknown fine-tuning examples, representing facts with lower degrees of certainty, plays an important part in properly handling such examples in test time. this indicates that the composition of fine-tuning examples significantly influences the extent to which llms effectively utilize their pre-existing knowledge. to summarize, we study the effect of new factual knowledge in the fine-tuning data by designing a controlled setup that isolates this factor. we find that fine-tuning examples that introduce new knowledge are learned slowly, which suggests that llms struggle to integrate new knowledge through finetuning and supports the view that llms mostly acquire knowledge through pre-training (zhou et al., 2023; lin et al., 2023). however, we also find that as the model eventually learns new knowledge through fine-tuning, it becomes more prone to hallucinations w.r.t. its pre-existing knowledge. collectively, our findings highlight the potential for unintended consequences when introducing new knowledge through fine-tuning, and imply that finetuning may be more useful as a mechanism to enhance the utilization of pre-existing knowledge. ## study setup given a fine-tuning dataset d and a pre-trained llm m , we denote by m d a model obtained by fine-tuning m on d. to study how new knowledge in d affects m d 's performance, we design a controlled setup creating variants of d with varying proportions of examples that are unknown to m . when constructing d, our objective is to reflect instruction tuning on diverse knowledge-intensive tasks while maintaining control over the experimental setting. we thus focus on factual knowledge that can be structured as (subject, relation, object) triplets, which are converted into closed-book qa format. in this setup, @xmath0, a i )} n @xmath1, where q is a knowledge-seeking question corresponding to a specific triplet (e.g., "where is paris located?") type category definition explanation ## quantifying knowledge in llms to assess the effect of new knowledge in d on the performance of m d , we have to annotate each (q, a) pair in d w.r.t. whether m knows that the answer to q is a. @xcite to estimate this, we define a continuous p correct measure based on samples from m , and use it to divide (q, a) pairs into four knowledge categories. we name this approach slick (sampling-based categorization of knowledge). defining p correct . we adopt the perspective that m knows that the answer to q is a if it generates a 3 palm-2 is available in five sizes: xxs, xs, s, m, l, with the s version representing the middle size in this range. @xcite we validated that in our setting em strongly correlates with word-level f1 (rajpurkar et al., 2016), and we choose em as it is more intuitive for the purposes of our analysis. @xcite we also considered using fake facts for introducing new knowledge, but we were concerned that this would introduce confounding factors into our study, as fake facts may behave differently than real ones. we discuss this in detail in §f. when prompted to answer q (kadavath et al., 2022; manakul et al., 2023). since m is a base model that has not been specifically fine-tuned to follow instructions, we prompt m using in-context learning with few-shot exemplars. following rubin et al. (2022), we make sure that the few-shot exemplars have high semantic similarity to q. @xcite in practice, m can predict different answers since (1) the choice of exemplars influences individual predictions and (2) temperature sampling, if used, introduces randomness. to reflect this, we define p correct (q, a; m, t ) as an estimate of how likely is m to accurately generate the correct answer a to q, when prompted with random few-shot exemplars and using decoding temperature t . for the purposes of our study we approximate the value of p correct using n @xmath2. @xcite for each 4-shot prompt, we predict the greedy answer using @xmath3.5. p correct (q, a; m, @xmath4, and p correct (q, a; m, @xmath5. full details are in §c. deriving knowledge categories from p correct . we define the unknown category (bottom row in figures 2a and 2b ) to represent (q, a) pairs for which m never predicts the correct answer to q. in our notations this means that p correct (q, a; m, @xmath6. alternatively, if p correct (q, a; m, @xmath7, i.e. m sometimes predicts the correct answer to q, we consider (q, a) as known. in this choice, we posit that if prompting m to answer q can sometimes result with the correct answer a, then m must have some association with the relevant fact. recognizing that knowledge can vary in degrees of certainty and extent, we divide the known (q, a) pairs into three distinct categories (top three rows in tables 2a and 2b ). motivated by the principle that m should consistently predict a if (q, a) is known, we put emphasis on greedy decoding outcomes, represented with p correct (q, a; m, @xmath8. highlyknown represents (q, a) pairs for which m always greedily predicts a. if m sometimes (but not always) greedily predicts a, we consider (q, a) as maybeknown. lastly, if m never greedily predicts a, we classify (q, a) as weaklyknown. we apply slick to annotate each (q, a) pair in our dataset with its knowledge category w.r.t. m . @xcite we analyze the quality of our categories in §6. in this section we study the effect of new knowledge in the fine-tuning dataset d on performance. to isolate this effect, we vary the proportion of unknown examples in d, while controlling for other factors. specifically, we fix |d| and create variants of d with x% of unknown and (100 -x)% known examples (full details in §e). we treat the known categories collectively (see figure 2a ), and provide a per-category analysis in §5. we de-note early-stopping based on the development set as early_stop (happens after 5-10 epochs) and 50 fine-tuning epochs as convergence, as at this point m always completely fits d (i.e. 100% training accuracy). we measure test performance as a proxy for hallucinations since we are in a closed-book qa setup with disjoint train/test splits, where the model has to use its per-existing knowledge to answer test questions (see §b for further discussion). ## understanding knowledge types: their value and impact when addressing our main research question on the effect of unknown fine-tuning examples, we treated the known categories collectively for simplicity (see figure 2a ). we now examine the effect of each category, exploring the following questions: q1: how training examples from each category impact the test performance? q2: what is the model's performance across test examples from each category? to address q1 we created single-category variants of the fine-tuning dataset d. a variant of d consisting solely of examples from the category cat is denoted as d cat . for reference, we include a variant with the natural categories distribution in entityquestions, denoted d natural . |d| is fixed and identical to our experiments in §4. to address q2, we further break down the test set performance by category. highlyknown test examples, yet its performance on the remaining categories is inferior. d maybeknown yields the best overall performance. compared to d highlyknown , d maybeknown enhances m d 's performance on maybeknown (60.1 → 69.9), without compromising performance on highlyknown (98.7 → 98.4). this suggests that maybeknown fine-tuning examples are essential for m d to correctly handle such examples during inference. it also demonstrates that with the right fine-tuning examples, m d becomes more capable of utilizing its pre-existing knowledge. limited knowledge enhances overfitting. in §4.2, we demonstrated that unknown fine-tuning examples increase the risk of overfitting. we now observe that this also applies to weaklyknown, though to a lesser degree. specifically, at convergence, d weaklyknown and d unknown experience significant performance drops compared to early_stop (39.2 → 35.4 and 37.5 → 25.8). with training to convergence, they show a modest improvement on weaklyknown and unknown but substantially degrade on highlyknown and maybeknown. this highlights that the decrease in performance is strongly attributed to an increased rate of hallucinations w.r.t. facts that were already known to m after pre-training. interestingly, d natural performs on-par with d maybeknown in early_stop, suggesting that the mere presence of maybeknown examples in d suffices for high performance on maybeknown, even if d has additional examples from other categories. yet, d natural 's performance degrades significantly 10 after convergence, under-performing d maybeknown -indicating that it still suffers from overfitting, most-likely due to the presence of weaklyknown and unknown examples. taken together these results demonstrate that d maybeknown stands out both in terms of top performance and reduced risk to overfitting. fine-grained known categories we first reflect on whether our choice of splitting known into more fine-grained categories, based on the greedy decoding outcome, has been proven meaningful. as shown in ## slick knowledge categories analysis as a case study for comparison, we analyze the p(true) approach by kadavath et al. (2022): a continuous score that estimates the probability a model assigns to the correctness of a specific answer. p(true) was originally used for self-evaluating model-generated answers, while we use it to assess whether m considers the ground-truth answer as correct. @xcite in figure 5 , we explore classifying examples below a p(true) threshold as unknown and compare this methodology to slick. our results indicate that, at least in our setting, our approach categorizes unknown examples for which the model's performance after fine-tuning is significantly worse. specifically, looking at fixed values on the x-axis shows that if we would label a similar fraction of test examples as unknown using both methods, the accuracy on the p(true)-based unknown examples would be much higher post fine-tuning. @xcite lastly, the blue line shows that using samples from multiple few-shot prompts to approximate p correct is crucial, as using n @xmath9. ## fine-tuning to abstain on unknown examples we showed that fitting unknown fine-tuning examples negatively affects the test performance ( §4.2 and §4.4). however, this negative effect manifests as a form of overfitting. from practical perspective, we showed that we can mitigate overfitting by either using early-stopping or filtering-out unknown examples from the fine-tuning dataset. we now explore an additional approach where we fine-tune the model to abstain from unknown examples as a potential mitigation. specifically, we replace the label of the unknown fine-tuning examples with the expression "i don't know" and test whether this mitigates the observed overfitting. ## discussion practical implications. this work highlights the risk in using supervised fine-tuning to update llms' knowledge, as we present empirical evidence that acquiring new knowledge through fine-tuning is correlated with hallucinations w.r.t pre-existing knowledge. additionally, this work raises important questions for future exploration regarding fine-tuning practices. we saw that unknown examples are fitted slower than the known ones, thus their negative effect manifests as a form of overfitting, which emphasizes the importance of using early-stopping instead of a fixed number of fine-tuning steps. however, earlystopping may be less effective when fine-tuning on numerous tasks with distinct optimal stopping points. an alternative solution can be to avoid adding new knowledge, by aligning the fine-tuning data with the model's knowledge through filteringout unknown examples. we show initial evidence that this can reduce the risk of overfitting without compromising performance. a possible drawback of filtering, is that unknown fine-tuning examples can still be useful to teach llms to express uncertainty on unknown test examples (zhang et al., 2023; yang et al., 2023). this raises the question: can re-labeling unknown fine-tuning examples with uncertainty expressions (e.g., "i don't know") reduce their negative effect? our experiment ( §7) superficial alignment hypothesis. zhou et al. (2023) hypothesized that the knowledge and capabilities of llms are mostly learned during pretraining, while alignment is a simple process where the model learns the style or format for interacting with users. they substantiate this hypothesis by showing that fine-tuning on just 1k high-quality examples can result with a competitive assistant llm, named lima. as discussed in §4.3, we show evidence that llms struggle to acquire new knowledge present in the unknown examples and mostly learn to utilize their pre-existing knowledge. we also showed that fine-tuning on highlyknown examples led to sub-optimal utilization of preexisting knowledge, despite our task format being simpler than lima's and our dataset being six times larger. taken together, our findings suggest that even though most of the llm's knowledge is indeed acquired through pre-training, the model learns more than just style or format through finetuning, as the selection of fine-tuning examples significantly influences the model's capability to utilize its pre-existing knowledge post fine-tuning. new knowledge and hallucinations. schulman (2023), goldberg (2023) and gudibande et al. (2023) mention the conjecture that fine-tuning on new factual knowledge may encourage hallucinations. huang et al. (2023) categorized hallucination causes and formally defined this scenario as capability misalignment, also highlighting that limited research addresses capability misalignment due to the challenge of defining the knowledge boundary of llms. recent work support our findings. for instance, ghosal et al. ( 2024 ) showed that models fine-tuned on well-known facts exhibit enhanced factuality compared to those fine-tuned on unpopular facts, which can be attributed to the model's lesser familiarity with unpopular facts. another example is lin et al. (2024), who fine-tuned a model using data generated by either a pre-trained model or a retrieval-augmented variant. they found that the latter resulted in reduced factuality, which can be attributed to the introduction of new factual knowledge in the retrieved texts. ren et al. (2024) have also investigated the effects of introducing new factual knowledge through fine-tuning in a considerably different methodological setup, focusing on multiple-choice questions, conducting relatively short fine-tuning runs, and testing only 100% known and 100% unknown mixtures. their results align with ours, which further reinforces our conclusions. lastly, these insights were also integrated into the instruction-tuning phase of llama 3 models (dubey et al., 2024), ensuring that the examples are aligned with pre-training knowledge. another line of work explores the model's behavior on new knowledge in test time. kang et al. (2024) showed that when a fine-tuned llm encounters unknown queries at test time, its responses mimic the responses associated with the unknown examples in the fine-tuning data. yin et al. (2023) showed that llms' performance is not satisfactory when they face new knowledge in their input contexts and lee et al. (2023) analyzed the impact of unknown in-context learning examples. quantifying knowledge in llms. slick can be seen as a confidence elicitation method for the ground truth label (m knows (q, a) if it is confident that a is the answer to q). existing work derive cali-brated confidence from llms by examining agreement across multiple samples (kuhn et al., 2023; manakul et al., 2023; tian et al., 2023a; lyu et al., 2024), probing internal representations (azaria and mitchell, 2023; burns et al., 2022), eliciting verbalized probability (tian et al., 2023b) or direct prompting (kadavath et al., 2022). kadavath et al. also trained a separate p(ik) model to predict if the llm knows the answer to q. the label for p(ik) was approximated by the fraction of correct sampled answers, which is conceptually aligned with p correct ( §3). a key difference is that we also define the slick categories, and provide evidence that we capture meaningful and useful categories. ## conclusion we study the impact of integrating new factual knowledge through fine-tuning on the model's tendency to hallucinate. we first propose slick, a categorization of facts w.r.t. llm's knowledge. we then design a controlled study where we isolate the impact of new knowledge and rigorously evaluate its effects. we provide multiple insights on the fine-tuning dynamics, with the following key findings: (1) acquiring new knowledge via supervised fine-tuning is correlated with hallucinations w.r.t. pre-existing knowledge. (2) llms struggle to integrate new knowledge through fine-tuning and mostly learn to use their pre-existing knowledge.