Paper: AutoHall: Automated Hallucination Dataset Generation for Large Language Models (ICLR-2024)

Reviewer comment (Novelty):
The authors place considerable emphasis on the model-specific nature of the automatically constructed hallucination datasets. It is unclear why the creation of such datasets is necessary, as they may not be applicable for evaluating other models due to their model-specific attributes. Furthermore, the paper does not propose a method for leveraging these datasets to mitigate the generation of hallucinatory content by the models.

Author response:
**C#1&Q#3: What is the underlying objective behind creating model-specific hallucination datasets, and how were these datasets applied in this study?**

We clarify the necessity of model-specific hallucination datasets in the **Sec.2** of our paper. Due to the scarcity of high-quality hallucination datasets, it is challenging to determine the specific types and degrees of hallucination exhibited by different LLMs. Existing approaches for hallucination detection often require manual annotation, which is time-consuming and labor-intensive.
Our objective is to provide a unified method for automatically collecting such datasets applicable to various models. These datasets serve as a benchmark for evaluating different LLMs and facilitate the development of techniques to detect and mitigate LLM hallucinations. Researchers can leverage our AutoHall to collect up-to-date model-specific hallucination datasets, even as models update.

**C#1: The paper does not propose a method for leveraging these datasets to mitigate the generation of hallucinatory content by the models.**

Hallucination mitigation is beyond the scope of this paper and our focus is automatic hallucination detection dataset construction and hallucination detection based on our constructed datasets.

Reviewer comment (Writing):
The paper lacks a clear and coherent explanation of the interrelation between the two methods presented, namely hallucination dataset construction and hallucination detection.

Author response:
**C#2&Q#1: Could you explain the relationship or connection between the two methods introduced in this study, namely, hallucination dataset construction and hallucination detection?**

Sure. We first need construct up-to-date hallucination datasets to find the varying characteristics of hallucination among different LLMs, which is the first part of our study. And then in the second part, we design a novel hallucination detection approach based on our constructed datasets.

Reviewer comment (Novelty):
The paper's reliance on the concept of self-contradictions as a novel approach for detecting hallucination in LLMs is questionable. Several prior studies have explored similar methodologies in hallucination detection, raising concerns about the novelty of the proposed method.

Author response:
**C#3: Concerns about the novelty of the proposed method.**

Our work makes a significant contribution in an automatic approach for generating hallucination detection datasets, which aligns with the title of our paper and has enough novelty. In addition, proposed hallucination detection method is another contribution. Although several prior studies utilize self-contradictions for detection, we employ the LLM itself to assess the presence of contradictions instead of relying on external tools (Manakul P et al., 2023). And unlike previous methods that compare the presence of contradictions among multiple sampled responses, we adopt a pairwise comparison approach to reduce false negatives.

Refs:

1. Manakul P, Liusie A, Gales M J F. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models[J]. arXiv preprint arXiv:2303.08896, 2023.

Reviewer comment (Writing):
Could you explain the relationship or connection between the two methods introduced in this study, namely, hallucination dataset construction and hallucination detection?

Author response:
**C#2&Q#1: Could you explain the relationship or connection between the two methods introduced in this study, namely, hallucination dataset construction and hallucination detection?**

Sure. We first need construct up-to-date hallucination datasets to find the varying characteristics of hallucination among different LLMs, which is the first part of our study. And then in the second part, we design a novel hallucination detection approach based on our constructed datasets.

Reviewer comment (Experiments):
What is the rationale behind relying on self-generated datasets for evaluating the efficacy of the proposed detection methods? Would it not be more compelling to employ other datasets for evaluation purposes?

Author response:
**Q#2: What is the rationale behind relying on self-generated datasets for evaluating the efficacy of the proposed detection methods? Would it not be more compelling to employ other datasets for evaluation purposes?**

Here are some reasons why we do not employ other datasets for evaluation:

1.	Applicability of existing hallucination datasets: Some of the publicly available hallucination datasets were collected from the previous version of LLMs such as GPT-3. However, these datasets are not suitable for evaluating the latest model which may produce different types or degrees of hallucination.

2.	Differences between task-specific datasets and our open-domain generation scenario: Some existing datasets are task-specific (such as for summarization or QA tasks), while our work focuses on hallucination detection in open-domain generation scenarios. Those datasets may not adequately cover the range of hallucinations that occur in open-domain generation contexts.

Reviewer comment (Evaluation):
One selling point of AutoHall is that it's automatic, but it requires relying claims and ground-truth evidence from fact checking datasets, which were curated by humans.

Author response:
**C#2: One selling point of AutoHall is that it's automatic, but it requires relying claims and ground-truth evidence from fact checking datasets, which were curated by humans.**

We politely disagree with your statement. Publicly available fact-checking datasets are already with labels, and our AutoHall process does not involve any manual annotation. Considering current hallucination benchmarks are highly dependent on human annotation (Li et al., 2023; Umapathi et al., 2023; Dale et al., 2023), our automatic AutoHall approach is undoubtedly meaningful.

Refs:

1. Li et al., Halueval: A large-scale hallucination evaluation benchmark for large language models. EMNLP 2023.

2. Umapathi et al., Med-halt: Medical domain hallucination test for large language models. EMNLP 2023.

3. Dale et al., HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation. 2023.

Reviewer comment (Evaluation):
The reported hallucination numbers, where Llama and ChatGPT produce similar hallucination rates, are very different than the numbers reported in https://arxiv.org/abs/2305.14251 where ChatGPT is nearly twice as factual as fine-tuned versions of Llama (though it's Llama 1 instead of 2, I'd be surprised if Llama 2 closes the gap), raising questions about the fidelity of the generated datasets.

Author response:
**C#3: The reported hallucination numbers, where Llama and ChatGPT produce similar hallucination rates, are very different than the numbers reported in https://arxiv.org/abs/2305.14251 where ChatGPT is nearly twice as factual as fine-tuned versions of Llama (though it's Llama 1 instead of 2, I'd be surprised if Llama 2 closes the gap), raising questions about the fidelity of the generated datasets.**

We have searched for some relevant works to demonstrate our conclusion. (Du et al., 2023) state that GPT-3.5-turbo has a hallucination rate of 26% in Fig. 3 which is consistent with ours. Meta claims (A.4.8, P69, Touvron et al., 2023) Llama-2-chat indeed improves a lot in terms of LLM hallucinations, which is also demonstrated in (Fig.1 & Tab. 1, Cheng et al., 2023). 

Refs:

1. Du et al., Quantifying and Attributing the Hallucination of Large Language Models via Association Analysis. 2023.

2. Touvron et al., Llama 2: Open foundation and fine-tuned chat models. 2023.

3. Cheng et al., Evaluating Hallucinations in Chinese Large Language Models. 2023.

Reviewer comment (Evaluation):
Why do you think there's the discrepency in the AutoHall conclusions and the factscore conclusions?

Author response:
**Q#1: Why do you think there's the discrepancy in the AutoHall conclusions and the factscore conclusions?**

Factscore evaluates traditional fact-checking tasks doing information retrieval on existing claims. However, hallucination detection emphasizes evaluation on the content generated by the models themselves, on which traditional fact-checking tasks pay little attention.
Besides, the FactScore evaluation is not equivalent to the hallucination rates concluded by our AutoHall. Fine-grained FactScore focuses on atomic facts which are more fundamental units than sentences, while the hallucination rate is a sentence-level metric. Therefore, the conclusions drawn from AutoHall and FactScore evaluations are not directly comparable due to the above differences.

Reviewer comment (Theory):
The method to classify the generated references is unreasonable since the two-phase pipeline brings much noise and accumulates errors. In Step 2, LLMs take the risk of producing wrong answers from factual references and vice versa.

Author response:
Numerous studies (Li et al., https://arxiv.org/abs/2305.14623; Cheung et al., https://arxiv.org/abs/2309.00240; Zhang et al., https://arxiv.org/abs/2304.03728) have demonstrated the high accuracy of LLM-based fact-checking approaches despite the potential for classification errors, among which Li et al. and Zhang et al. also used LLMs with instructions like ours. 
We further conduct an additional experiment by randomly selecting 100 (claim, reference) pairs (**dataset: Climate-fever, model: ChatGPT, temperature: 0.9**) and manually assessing whether the classification results are correct. The results show that the LLM classification accuracy reaches **92%**.

Reviewer comment (Experiments):
The problem caused by model randomness in the data generation process has not been considered. For example, for a given (claim, reference) pair, the LLM’s label prediction could be influenced by sampling-based decoding and the form of instructions, as LLMs are sensitive to different instructions (Zhou et al., 2023).

Author response:
In our study, we carefully design and experiment with the prompts used for classification, and the prompt ultimately selected prove to be effective across different models. 
We also conduct additional experiments using **six** prompt variants (from simple (P0) to complex (P5)) to assess the classification performance on prompt sensitivity. Results show that there is no significant correlation between the prompt complexity and LLMs’ classification accuracy. 
| Prompts | P0 | P1 | P2 | P3 | P4 | P5 |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Acc.(%) | 94.0 | 93.6 | 92.8 | 93.9 | 92.6 | 93.1|

Reviewer comment (Experiments):
The necessity of model-specific generation has not been verified. This motivation makes sense, but the authors should also justify it by testing the transferability of datasets generated by different models.

Author response:
The transferability of datasets generated by different models is beyond the scope of our study and it is improper to test whether other models have hallucinations using data generated by a specific model. It is important to note that our research aims to automatically construct hallucination datasets and explore what types or topics of LLM responses that tend to be hallucinatory.

Reviewer comment (Novelty):
The method of automatically generating hallucinated references was proposed by (Agrawal et al., 2023). The only difference lies in that the authors generated references from existing fact-checking claims instead of topic words.

Author response:
Not really. Our AutoHall approach centers on claim classification based on the idea that generated hallucinatory references lead to misclassification of claims since LLMs are excellent classifiers (Stoliar et al., Using LLM classification in foresight studies). This is totally different from direct queries in (Agrawal et al., 2023) by prompting the LM to check the existence of the reference.

Reviewer comment (Novelty):
The proposed self-contradiction-based hallucination detection method has also been proposed in previous work (Mündler et al., 2023; Manakul et al., 2023; Cohen et al., 2023), among which Mündler et al., and Cohen et al., both utilized LLMs as the contradiction judge. Besides, the design of self-contradiction detection in $(Y,Y^{'}_k)$ pair seems meaningless. Since the generation process of $Y$ is the same as that of $Y^{'}_k$, $Y$ can be considered one of $Y^{'}_k$. The claim "*avoid the problem that SelfCheckGPT incorrectly identifies the conflicts in the K sentences generated*" was not verified.

Author response:
Not really. Self-contradiction detection in $(Y,Y_{k}^{'})$ pair is meaningful as it avoids the situation that detecting conflicts together like SelfCheckGPT incorrectly labels hallucination in Y when in fact hallucinations exist among K sentences. We admit that there is still a possibility of hallucination existing in $Y^{'}_{k}$ while not in Y which increases false positive to some extent. However, by doing self-contradiction detection in pairs, we can find almost only when hallucinations exist in Y can the number of conflicts exceed half from Fig. 6. Thus, our approach in general achieves higher F1 score than SelfCheckGPT as is shown in our experiments. And we wrote the legend text in reverse order by mistake in Fig. 6. We will revise it in the later version.

Reviewer comment (Experiments):
Some essential baselines are not compared. For example, (Mündler et al., 2023), which also uses LLMs as the contradiction judge in an NLI manner.

Author response:
Not really. Although the work (Mündler et al., 2023) also uses self-contradictions for detection, they are dependent on context (blue text named description in Fig. 2) which is different from our open-domain generation scenarios without any context guided.

Reviewer comment (Evaluation):
It is unclear why self-check performed worse in the few-shot setting than the zero-shot setting with ChatGPT in Table 3.

Author response:
a)	The experimental results in (Tab. 5, Chern et al., https://arxiv.org/abs/2307.13528) also use the same Self-Check baselines and show the same comparison results with ours. We suppose that CoT mechanism may not perform its best in open-domain generation scenarios, thus few-shot or zero-shot setting have little impact. So regardless of the types of LLMs, Self-Check baselines are prone to false positive and perform worse.

Reviewer comment (Evaluation):
It is unclear why 1gm was worse than the few-shot with LLaMa2-7B.

Author response:
b)	The performance of SelfCk-1gm relies highly on the variety of LLM responses. Since the number of parameters in a model determines its language understanding and generation capabilities (Tab.3, Touvron et al., https://arxiv.org/abs/2307.09288), the 7B model may be the weakest in performance among baselines. Less world knowledge, less variety in responses and more difficult to detect inconsistency, thus SelfCk-1gm with LLaMa2-7B may be worse than the few-shot while with other baselines are the opposite.

Reviewer comment (Presentation):
Fig. 1 is blurry.

Author response:
Thanks for pointing this detail out. We will revise it in the later version.

Reviewer comment (Presentation):
It is unclear whether Fig. 6 is the opposite of the conclusion, as it seems to suggest that the more conflicts there are, the more factual references.

Author response:
Not really. Self-contradiction detection in $(Y,Y_{k}^{'})$ pair is meaningful as it avoids the situation that detecting conflicts together like SelfCheckGPT incorrectly labels hallucination in Y when in fact hallucinations exist among K sentences. We admit that there is still a possibility of hallucination existing in $Y^{'}_{k}$ while not in Y which increases false positive to some extent. However, by doing self-contradiction detection in pairs, we can find almost only when hallucinations exist in Y can the number of conflicts exceed half from Fig. 6. Thus, our approach in general achieves higher F1 score than SelfCheckGPT as is shown in our experiments. And we wrote the legend text in reverse order by mistake in Fig. 6. We will revise it in the later version.