Paper: Teaching Language Models to Hallucinate Less with Synthetic Tasks (ICLR-2024)

Reviewer comment (Experiments):
The evaluation of SYNTRA is somewhat limited, as it focuses on only 2 models and 3 realistic tasks. This raises concerns about the method's ability to generalize across diverse models and tasks. Also, the effect on Vicuna seems marginal. This weak effect could limit the practical utility of SYNTRA, especially if similar results persist across other models.

Author response:
_The evaluation of SYNTRA is somewhat limited, as it focuses on only 2 models and 3 realistic tasks. This raises concerns about the method's ability to generalize across diverse models and tasks._

Thanks for raising this point; based on this feedback, we ran SynTra on Llama 2 7B chat and Llama 2 13B chat, both with RLHF.  We find that SynTra manages to reduce hallucination in both models under these datasets in a comparable way to the existing models. We include full results in Appendix B.8, but find substantial improvements; for example, SynTra doubles the BLEU score for MS MARCO 13B.

Reviewer comment (Presentation):
The way the results are presented can be improved. There are some key takeaway messages that are hidden in the table/figure. Important insights, such as the significant reduction in grounded entities generated by Orca and Vicuna (by 66% and 99% respectively) when using SYNTRA without reference data, are buried within tables and figures.

Author response:
_Important insights, such as the significant reduction in grounded entities generated by Orca and Vicuna (by 66% and 99% respectively) when using SYNTRA without reference data, are buried within tables and figures_

Sorry for the confusion; **the 66% and 99% reduction in entities are for the fine-tuned model, not the system message**: the numbers for the system message are (1.4% and 14.8% respectively). We also find that this reduction is small for the Llama 2 models, (e.g., the reduction for Llama 2 7B is only 6.2%). Moreover, this reduction is consistently less than the reduction in ungrounded entities, which are what we aim to remove. We hope this assuages your concern, and if it changes your impression of our results that you’ll consider increasing your score.

Reviewer comment (Evaluation):
The observation that LLMs, in their attempt to avoid hallucination, compromise by producing limited details should be more prominently highlighted, especially to stress that this setup may not be a viable solution for hallucination reduction.

Author response:
> I understand that the 66% and 99% reduction in entities are for the fine-tuned model, not the system message. My point in the original review is exactly regarding the fine-tuned approach. If we see this much of reduction in entities for the fine-tuned model, then it should be prominently highlighted that users should also be cautious of this approach (if not never use this) and again stresses that this fine tuning setup may not be a viable solution for hallucination reduction.

That makes sense, thanks for clarifying this! The 66% and 99% numbers are a baseline within a baseline: they come from training on just the names data (without mixed in reference) using fine-tuning (instead of tuning the system message). The numbers with just fine-tuning are 12.2% and 17.6% for Orca and Vicuna respectively (Table 3). These are nonzero, but fine-tuning reduces the number of "ungrounded" entities by much more: 36.5% and 43.2% respectively, which might be an ok tradeoff for practitioners. We're space constrained with the edits we can make in the rebuttal, but will add a discussion about how (i) fine-tuning lowers the entity counts by more (i.e., beyond the "Fine-tuning and training without the reference data eliminate more grounded and ungrounded entities" that is already there) and (ii) fine-tuning without reference data can be degenerate to the discussion in 4.3. We can also preview this in the appendix if it's helpful, and overall think this largely enhances the message by adding additional evidence that optimizing the system message transfers better.

Reviewer comment (Presentation):
The axis label "Hallucination rate" in Figure 2 is misleading. The number presented therein should instead be 1-hallucination rate, unless there is a misunderstanding.

Author response:
_I find the axis label "Hallucination rate" misleading in Figure 2 - I think the number presented therein should instead be 1-hallucination rate? or did I miss anything?_

Yes you are right; thanks for raising this! We’ve updated the figure.

Reviewer comment (Experiments):
Can SYNTRA be added on top of other classes of methods mentioned in section 2 to further reduce hallucination?

Author response:
_Can SYNTRA be added on top of other classes of methods mentioned in section 2 to further reduce hallucination?_

This is possible; SynTra already reduces hallucination on top of instruction tuning (for Vicuna and Orca), which already reduced hallucination. We believe this is an exciting direction for subsequent work.

Reviewer comment (Theory):
The authors mentioned at the end of the Related Work that the direct optimization of a discrete prompt might be viable in SYNTRA as well. Can they give more discussions about the pros and cons compared to SYNTRA? Will it be more difficult in terms of the optimization?

Author response:
_The authors mentioned at the end of the Related Work that the direct optimization of a discrete prompt might be viable in SYNTRA as well, can they give more discussions about the pros and cons compared to SYNTRA? Will it be more difficult in terms of the optimization?_

Yes, doing the discrete optimization is likely harder (e.g., see https://arxiv.org/abs/2306.15447 which shows that even the best discrete optimizers over tokens struggle), but may produce more in-distribution system messages that generalize more. We have added a note about this to the discussion.

Reviewer comment (Experiments):
From Figure 2 and Table 1, it looks like when LLM weights are fine-tuned, they are overly optimized for the synthetic task and as such it might end up hallucinating more on realistic tasks. Is there any possibility that an early-stopping of the fine-tuning process might actually help in finding better parameters for both synthetic and realistic tasks?

Author response:
_From Figure 2 and Table 1, it looks like when LLM weights are fine-tuned, they are overly optimized for the synthetic task and as such it might ends up hallucinate more on realistic tasks. Is there any possibility that an early-stopping of the fine-tuning process might actually help in finding better parameters for both synthetic and realistic tasks?_


This is possible; there are ways to both improve fine-tuning and postfix-tuning. Nevertheless, the broader paradigm of supervising on known tasks then transferring is a promising approach, and developing better ways to transfer (e.g., other fine-tuning methods or beyond) are promising future directions.

Reviewer comment (Experiments):
Lack of baselines: The proposed method was applied to existing fine-tuned LLMs like Vicuna and Orca and the experiments only show that they become better than the original LLMs. However, we do not know how the quality of the SynTra task compared to other datasets. For example, a more fair comparison would be finetuning two LLMs starting from LLaMA, one on the Vicuna ShareGPT data, the other on the SynTra data. I wonder if the effects of the SynTra data are not as much as the Vicuna data, in terms of reducing hallucinations.

Author response:
Thanks for your review and comments! We’re glad you appreciated how our method is transferable and novel. 

It seems like your main concern is that the reduction in hallucination rate is due to the ShareGPT / original LLM fine-tuning set, rather than SynTra. **We think this is a misunderstanding: we compare SynTra trained on top of the fine-tuned models to the fine-tuned models themselves, not to Llama**. In particular, we show that SynTra reduces hallucination _on top of_ any hallucination reduction obtained by the original fine-tuning. This highlights the complementary nature of SynTra to other approaches; SynTra further reduces hallucination after fine-tuning. We updated all tables to say e.g., “Original Orca”, instead of “Original Model” to emphasize this and hope that if this and our other responses help resolve your concerns, you’ll consider increasing your score.

Reviewer comment (Experiments):
Comparison with the code datasets: [previous work](https://arxiv.org/abs/2210.07128) [1] has found that training the model on code datasets is beneficial for reasoning. I wonder how the quality of the SynTra task compared to directly fine-tuning the model on the code dataset?

Author response:
_Comparison with the code datasets: previous work [1] has found that training the model on code datasets is beneficial for reasoning. I wonder how the quality of the SynTra task compared to directly fine-tuning the model on the code dataset?_

We haven’t tried fine-tuning on code datasets; it’s possible this would help, although reasoning ability may not directly reduce hallucination. However, we expect that SynTra may be complementary to this approach; SynTra reduces hallucination further after fine-tuning on high-quality GPT-4 generated data for Orca, suggesting it improves performance even on fine-tuned models.

Reviewer comment (Experiments):
In general, training on the rule-based synthetic data would help is expected. However, the question that matters is how useful this data could be compared to other existing datasets such as instruction tuning datasets or code datasets. I suspect that the effects of synthetic data would be marginal when compared to others.

Author response:
Thanks so much for getting back to us and for your thoughtful comment! It is indeed probably also the case that the ShareGPT / Orca fine-tuning reduces a lot of hallucinations (and instruction tuning in general is probably required for these tasks, which require cohesive long-form outputs). We didn't run the hallucination evaluation on llama 1 because it seems to struggle with our tasks without instruction tuning, but think exploring the comparative benefits of synthetic and real data on top of the instruction tuned models is interesting, and exciting for future work. The last paragraph in our discussion section discusses the tradeoffs between synthetic data and demonstrations, and we've updated it based on your feedback (see new revision).

Thanks again for engaging in the rebuttal, and please reach out if any other questions come up!

Reviewer comment (Evaluation):
The transfer from synthetic to downstream tasks could be brittle: Sometimes full fine-tuning is better than soft-prompt tuning in system messages, but sometimes it's not. Sometimes reference data regularization is helping, sometimes it's not.

Author response:
_The transfer from synthetic to downstream tasks could be brittle: Sometimes full fine-tuning is better than soft-prompt tuning in system messages, but sometimes it's not. Sometimes reference data regularization is helping, sometimes it's not._

This is true; for specific datasets some approaches work better than others (e.g., training without the reference data is better on MS MARCO). We think this is a byproduct of transferring the same model weights / system message across multiple diverse tasks, and note that on average, training on the system message + regularization data is best for both models.

Reviewer comment (Presentation):
In figure 2, the y-axis says hallucination rate (the lower the better), but according to the passages, the y-axis should be the accuracy (the higher is better). Please clarify this.

Author response:
_In figure 2, the y-axis says hallucination rate (the lower the better), but according to the passages, the y-axis should be the accuracy (the higher is better). Please clarify this._

Thanks for bringing this up! We updated the figure accordingly.

Reviewer comment (Writing):
Citation error: (yew Lin & Rey, 2004) -> (Lin & Rey, 2004)

Author response:
_Citation error: (yew Lin & Rey, 2004) -> (Lin & Rey, 2004)_

Fixed, thank you!

Reviewer comment (Experiments):
The authors only experiments with one model size (13B) and it’s hard to say how well their method scales. Will be benefits of SynTra increase or decrease for finetuned LLaMA 30B or 65B? What about LLaMA 6.5B?

Author response:
_The authors only experiments with one model size (13B) and it’s hard to say how well their method scales. Will be benefits of SynTra increase or decrease for finetuned LLaMA 30B or 65B? What about LLaMA 6.5B?_

SynTra performs comparably on Llama 2 7B chat and Llama 2 13B chat. We do think that SynTra provides an interesting alternative approach to human feedback, in the limit where human feedback becomes harder to elicit (e.g., in higher quality outputs, humans may increasingly struggle to identify hallucinations).

Reviewer comment (Experiments):
It was a mildly surprising finding for me that soft system message optimisation is so much better than supervised finetuning. Have you tried manipulating the $\alpha$ hyperparameter to increase regularisation for supervised finetuning? I could imagine this being helpful.

Author response:
_It was a mildly surprising finding for me that soft system message optimisation is so much better than supervised finetuning. Have you tried manipulating the $\alpha$ hyperparameter to increase regularisation for supervised finetuning? I could imagine this being helpful_

This is a good question; we only test two values for $\alpha$ (either 1 and 0.5), but it is possible that this, or other interventions, might improve fine-tuning (just as similarly, regularization + optimization \alpha could help SynTra). Intuitively, we expect that the system message helps because it optimizes over “high-level instructions’ that are likely to transfer, instead of actually changing the model mechanism. However, we note that better fine-tuning might help (see second paragraph of section 5), and think this is an interesting direction for subsequent work.

Reviewer comment (Presentation):
It’s confusing that “hallucination rate” in Fig. 2 is actually non-hallucination rate (higher is better) while for “hallucination rate” in Table 1 lower is better.

Author response:
_It’s confusing that “hallucination rate” in Fig. 2 is actually non-hallucination rate (higher is better) while for “hallucination rate” in Table 1 lower is better._

Updated the Figure: thanks for raising this!