Paper: Fine-Tuning Language Models for Factuality (ICLR-2024)

Reviewer comment (Novelty):
Although existing work may have not used a factuality-based reward, the results in this paper are mostly the expected observations (e.g., applying RL-based training improves target rewards). For example, (Lu et al., 2022) applied RL (PPO) with a reward based on an external metric to improve toxicity, repetition, etc.

Author response:
**Regarding novelty.** While our work indeed leverages DPO, an existing algorithm, usage of model confidences as a proxy for a learning signal for RL is to our knowledge completely novel, and we show this technique is very useful for improving factuality. A priori, it is not obvious that LLMs contain a strong enough internal model of what is true and false that they would be able to learn to increase their factuality in a generalizable way. That is, the novelty of our findings is that **we can directly fine-tune language models to improve their factuality from tractably-sized datasets, without human labels or test-time retrieval/repeated sampling.** Further, we also note the call for papers lists societal considerations including "fairness, safety, privacy" as specifically relevant topics to ICLR; in light of the important reliability and safety risks of language model hallucinations, we feel that our work showing substantial mitigation of this problem is highly relevant and will be of strong interest to the ICLR community.

**Re: additional literature on RL.** Thank you for this suggestion! To summarize, Lu et al., 2022 discuss an alternative approach to RL for language models (though it requires drawing samples during the training procedure), but does not discuss factuality. Manakul et al and Mundler et al describe other approaches to deriving reference-free truthfulness scores, which are compatible with our framework for factuality tuning; in contrast, our work is focused on showing that optimizing with RL even simple approaches to truthfulness scoring can lead to substantial reductions in hallucination. We have added more discussion of alternative RL algorithms in the preliminaries, discussion of the compatibility of alternative truthfulness scores in the related work, and the possibility of alternative approaches to preference construction through self-correction in the future work.

Reviewer comment (Evaluation):
How does the DPO fine-tuned model perform on out-of-domain tasks? For example, when fine-tuning to improve factuality on biographies, does it also improve factuality on medical QA? And does its general performance change? There is also a recent survey paper (Pan et al., 2023) about aligning LLMs for different aspects (including hallucination/factuality), and I think it would be useful for authors to incorporate additional relevant papers (i.e., those that apply RL to improve LMs).

Author response:
**Re: generalization across domains.** In a new experiment, we show that factuality tuning generalizes very well across data distributions. We evaluate the result of training on one data distribution (e.g, bios) and evaluating on a completely different one (e.g., medQA). **In summary, factuality tuning on bios and evaluating on medQA (or vice versa) increases factuality more than RLHF, indicating that factuality tuning makes a meaningfully general improvement to the model's factuality.**

Eval on Bios
| Method | Correct | Incorrect | % correct |
| --- | --- | --- | --- |
| RLHF | 19.0 | 6.41 | 0.748 |
| FactTune-FS | 21.0 | 4.50 | 0.824 |
| (OOD-med) FactTune-FS | 21.0 | 5.31 | 0.799 |

Eval on MedQA
| Method | Correct | Incorrect | % correct |
| --- | --- | --- | --- |
| RLHF | 9.63 | 5.50 | 0.636 |
| FactTune-FS | 9.50 | 5.63 | 0.680 |
| (OOD-bio) FactTune-FS | 8.81 | 5.12 | 0.658 |

**Re: factuality's impact on general sample quality.** To check for decreased capabilities, we measure the fluency of each model (fine-tuned from Llama-7b on biographies) compared to SFT, using GPT-4 as a judge. We find that **our approach leaves fluency of generated text essentially unchanged.** Further, DoLA harms model sample fluency (often through increased repetition).

- DOLA: Preferred 34%
- FactTune-FS (ours): Preferred 50%
- FactTune-MS (ours): Preferred 48%

**Re: additional literature on RL.** Thank you for this suggestion! To summarize, Lu et al., 2022 discuss an alternative approach to RL for language models (though it requires drawing samples during the training procedure), but does not discuss factuality. Manakul et al and Mundler et al describe other approaches to deriving reference-free truthfulness scores, which are compatible with our framework for factuality tuning; in contrast, our work is focused on showing that optimizing with RL even simple approaches to truthfulness scoring can lead to substantial reductions in hallucination. We have added more discussion of alternative RL algorithms in the preliminaries, discussion of the compatibility of alternative truthfulness scores in the related work, and the possibility of alternative approaches to preference construction through self-correction in the future work.

Reviewer comment (Experiments):
Weak base LM: This work uses Llama-7B as the base model, and at this size, the model may not yet be highly capable of long-form generation / medical QA. Previous works such as (Manakul et al., 2023) and (Mundler et al., 2023) investigated LLM hallucination with much larger LLMs (e.g., GPT3.5/4). It would be interesting to see, for example, when using larger models (either open-source such as larger Llama / Falcon-180B or private ones such as GPT-4), if the model still makes as many factual errors (because if they don’t – due to the emergent ability when scaling up – fine-tuning may not be necessary or have little impact). There is a recent survey paper (Pan et al., 2023) about aligning LLMs for different aspects (including hallucination/factuality), and I think it would be useful for authors to incorporate additional relevant papers (i.e., those that apply RL to improve LMs).

Author response:
**Re: strength of the base LM.** We repeated our experiments for a larger 30B Llama model, finding that the model still hallucinates quite often and that factuality tuning still provides reduction in errors.

- SFT:                11.4 correct, 6.48 incorrect, 0.638 accuracy 
- FactTune-FS: 12.5 correct, 5.84 incorrect, 0.693 accuracy

**Re: additional literature on RL.** Thank you for this suggestion! To summarize, Lu et al., 2022 discuss an alternative approach to RL for language models (though it requires drawing samples during the training procedure), but does not discuss factuality. Manakul et al and Mundler et al describe other approaches to deriving reference-free truthfulness scores, which are compatible with our framework for factuality tuning; in contrast, our work is focused on showing that optimizing with RL even simple approaches to truthfulness scoring can lead to substantial reductions in hallucination. We have added more discussion of alternative RL algorithms in the preliminaries, discussion of the compatibility of alternative truthfulness scores in the related work, and the possibility of alternative approaches to preference construction through self-correction in the future work.

Reviewer comment (Evaluation):
The test sets (50 and 59 examples in each domain, respectively) look very limited, making it a bit hard to understand the actual improvement of model factuality. How reliable are the results? Is 75% -> 81% a lot? I can't really answer these questions after reading the paper.

Author response:
**Regarding evaluation set size.** We generated newer, larger test sets using the same methodology as the originals. On these larger held out sets of 300 medical questions and 300 biography prompts, we find similar results as in our original test sets:

On 300 medical questions & Llama2: 
| Method | Correct | Total | % Correct |
| --- | --- | --- | --- |
| SFT | 9.16 | 15.9 | 0.582 |
| FactTune-FS | 12.6 | 16.2 | 0.783 |
| FactTune-MC | 11.3 | 16.9 | 0.671 |
| RLHF | 8.68 | 16.2 | 0.559 |

On 300 biographies & Llama2: 
| Method | Correct | Incorrect | % Correct |
| --- | --- | --- | --- |
| SFT | 12.3 | 7.10 | 0.648 |
| FactTune-FS | 16.0 | 3.13 | 0.841 |
| FactTune-MC | 11.5 | 3.40 | 0.783 |
| RLHF | 20.6 | 6.81 | 0.746 |

The relative performance of our methods remains similar – FactTune still surpasses SFT and RLHF. 


**Gaining intuition about the results.** Another way to gain intuition about the results is considering error rate: an improvement of 75% to 81% correctness corresponds to a reduction in error rate from 25% to 19%. In other words, **the fraction of a model's response that is factually incorrect has decreased by 24%.** Our Llama-2 experiments show even more substantial improvement of a **50% reduction in hallucinations for biography generation and answering medical questions** on the extended test set.

Reviewer comment (Evaluation):
There is no evidence indicating that the improved factuality doesn't come at the expense of performance in other aspects. I understand the authors may not have enough labor/compute for a more comprehensive evaluation like GPT or Llama, but LLMs, in my experience, can behave in mysterious ways when you over-index on one specific objective.

Author response:
**Impact of factuality tuning on general model sample quality.** As one way to check for decreased capabilities, we measure the fluency of each model (fine-tuned from Llama-7b on biographies) compared to SFT, using GPT-4 as a judge. We find that, for example, DoLA harms model sample fluency (often through increased repetition), while our approach leaves fluency essentially unchanged.

- DOLA: Preferred 34%
- FactTune-FS (ours): Preferred 50%
- FactTune-MS (ours): Preferred 48%

**Regarding the more general point of reward overoptimization.** While our previous two points have shown that factuality tuning does not come through exploiting length of responses or compromising the fluency of model samples, we also show that we do not over-exploit the training data by training on one data distribution (e.g, bios) and evaluating on a completely different one (e.g., medQA). **In summary, factuality tuning on bios and evaluating on medQA (or vice versa) increases factuality more than RLHF, indicating that factuality tuning makes a meaningfully general improvement to the model's factuality.**


Eval on Bios
| Method | Correct | Incorrect | % correct |
| --- | --- | --- | --- |
| RLHF | 19.0 | 6.41 | 0.748 |
| FactTune-FS | 21.0 | 4.50 | 0.824 |
| (OOD-med) FactTune-FS | 21.0 | 5.31 | 0.799 |

Eval on MedQA
| Method | Correct | Incorrect | % correct |
| --- | --- | --- | --- |
| RLHF | 9.63 | 5.50 | 0.636 |
| FactTune-FS | 9.50 | 5.63 | 0.680 |
| (OOD-bio) FactTune-FS | 8.81 | 5.12 | 0.658 |

Reviewer comment (Novelty):
The method is somewhat straightforward, which is not necessarily a bad thing if the evaluation can show meaningful improvements (that are worth fine-tuning specifically for factuality) compared to methods that modify decoding only.

Author response:
**Regarding novelty.** While our work indeed leverages DPO, an existing algorithm, usage of model confidences as a proxy for a learning signal for RL is to our knowledge completely novel, and we show this technique is very useful for improving factuality. A priori, it is not obvious that LLMs contain a strong enough internal model of what is true and false that they would be able to learn to increase their factuality in a generalizable way. That is, the novelty of our findings is that **we can directly fine-tune language models to improve their factuality from tractably-sized datasets, without human labels or test-time retrieval/repeated sampling.**  Further, we also note the call for papers lists societal considerations including "fairness, safety, privacy" as specifically relevant topics to ICLR; in light of the important reliability and safety risks of language model hallucinations, we feel that our work showing substantial mitigation of this problem is highly relevant and will be of strong interest to the ICLR community.

Reviewer comment (Evaluation):
It is unclear why the largest bin is chosen to measure truthfulness in the reference-free setting. Does that mean the atomic claim doesn't really matter ("Yo-Yo Ma was born in 1951" and "Yo-Yo Ma was born in 1955" would both be converted to "What year was Yo-Yo Ma born")? So as long as two responses make a claim on the same fact, regardless of whether it is correct or wrong, they will receive the same truthfulness score? If the hypothesis is "a language model’s confidence in a generated answer is highly correlated with the probability that the answer is correct", why not use the distribution to cross-check like in the reference-based setting?

Author response:
**Regarding question on reference-free preference construction.** The intuition behind choosing the confidence of the max bin, rather than the sampled bin, is to encourage elicitation of facts that the model is confident in (even if the model gets unlucky and samples a low-probability false completion in some samples). The confidence of the max bin is highly correlated with entropy over the sampling distribution (>0.95) but more interpretable.

Additionally, we expect the confidence of the max bin & the confidence of the sampled bin to be correlated as the max frequency option is most likely to be chosen. After computing the confidence of the sampled bin we found a 0.57 correlation between the scores.

We hope this helps answer your questions about our work!

Reviewer comment (Novelty):
Idea itself is not novel, RLAIF has been consistently shown to be useful (here, authors used DPO instead of PPO). Though the application is new.

Author response:
**Regarding novelty.** While our work indeed leverages DPO, an existing algorithm, usage of model confidences as a proxy for a learning signal for RL is to our knowledge completely novel, and we show this technique is very useful for improving factuality. A priori, it is not obvious that LLMs contain a strong enough internal model of what is true and false that they would be able to learn to increase their factuality in a generalizable way. That is, the novelty of our findings is that we can directly fine-tune language models to improve their factuality from tractably-sized datasets, without human labels or test-time retrieval/repeated sampling. Further, we also note the call for papers lists societal considerations including "fairness, safety, privacy" as specifically relevant topics to ICLR; in light of the important reliability and safety risks of language model hallucinations, we feel that our work showing substantial mitigation of this problem is highly relevant and will be of strong interest to the ICLR community.

Reviewer comment (Experiments):
Including more fine-tuning based baselines can help understand the role of automated metrics. E.g., directly using prompts to compare factuality of two outputs w.r.t. the wikipedia article instead of extracting atomic facts.

Author response:
**Regarding other fine-tuning baselines.** We asked GPT-3.5 to provide factuality preference pairs over our responses with CoT, but found that these preference pairs agreed with the FactScore-baed preference pairs only 55% of the time, suggesting that naively prompting GPT-3.5 does not have strong enough factuality signal.

Reviewer comment (Evaluation):
Could you provide statistics on the number of tokens in wining vs losing samples in all cases (dataset/model/metric)?

Author response:
Lastly, here are some length statistics:
Average # words/sample in train set: 100 words 
Fact Score: 
- Avg # words in winning vs losing samples: 96 vs 105 words  
- Avg difference in each pair: winning has 9.4 more words than losing 
Model Confidence: 
- Avg # words in winning vs losing samples: 96 vs 105 words  
- Avg difference in each pair: winning has 8.7 fewer words than losing 

Average # facts/sample in train set: 24 
Fact Score:
- Avg # facts in winning vs losing samples: 22.6 vs 25.1
- Avg diff (facts) in each pair: winning has 2.4 fewer facts than losing 
Model Confidence:
- Avg # facts in winning vs losing samples: 23.0 vs 24.7
- Avg diff (facts) in each pair: winning has 1.7 fewer facts than losing

Reviewer comment (Novelty):
Because the framework is simple and the method of scoring truthfulness and fine-tuning technique uses existing approaches, the proposed method appears to have limited contributions

Author response:
**Regarding novelty.** While our work indeed leverages DPO, an existing algorithm, usage of model confidences as a proxy for a learning signal for RL is to our knowledge completely novel, and we show this technique is very useful for improving factuality. A priori, it is not obvious that LLMs contain a strong enough internal model of what is true and false that they would be able to learn to increase their factuality in a generalizable way. That is, the novelty of our findings is that **we can directly fine-tune language models to improve their factuality from tractably-sized datasets, without human labels or test-time retrieval/repeated sampling.** Further, we also note the call for papers lists societal considerations including "fairness, safety, privacy" as specifically relevant topics to ICLR; in light of the important reliability and safety risks of language model hallucinations, we feel that our work showing substantial mitigation of this problem is highly relevant and will be of strong interest to the ICLR community.

Reviewer comment (Experiments):
In Biographies and Medical QA tasks, are DPO-FS, DPO-MC, and SFT fine-tuned on the training set of each dataset?

Author response:
**Train sets.** DPO-FS, DPO-MC, and SFT are all fine-tuned with the train set of each dataset.