Paper: Can LLM-Generated Misinformation Be Detected? (ICLR-2024)

Reviewer comment (Experiments):
The paper’s evaluation is done on a very small scale, particularly 100 pieces of news, and leveraging only 10 human evaluators to assess the performance of humans and compare it with various LLM-based automated detectors. Due to this, there is a concern about how robust and generalizable the presented results are.

Author response:
We conduct a much more extensive investigation on misinformation generated by ChatGPT, Llama-7b-chat-generated misinformation (or 13b, 70b) and Vicuna-7b-generated misinformation (or 13b, 33b) to further validate our findings. For each LLM as the misinformation generator, we compare the detection difficulty of different misinformation generation methods including Paraphrase Generation, Rewriting Generation, Open-ended Generation and human-written misinformation. We include multiple human-written misinformation datasets including Politifact, Gossipcop and CoAID. The results are in Appendix A.

We will expand the human evaluation part in our future work.

Reviewer comment (Evaluation):
The paper does not discuss whether the presented results and differences between human-written and LLM-generated misinformation are statistically significant. The results and differences are presented without providing any additional context on their statistical significance. It is suggested to expand the evaluation and discuss the statistical significance of the results.

Author response:
We carefully discuss the statistical significance of all the results in Appendix B.

Reviewer comment (Experiments):
The paper lacks important details on how the ten human evaluators are selected. It is unclear whether these evaluators have previous experience with annotating pieces of information as misinformation.

Author response:
We do not require the annotating experience. We ask evaluators to select a label of “factual” or “nonfactual” for each news item only based on their own first perceptions upon reading it. For each evaluator, the news articles are presented in a random order.

Reviewer comment (Experiments):
It is unclear if the authors took any steps to ensure that the annotations by human evaluators are of high quality and that the annotators did not use LLMs to solve the task. The paper by Veselovsky et al. (2023) showed that crowd workers are using LLMs to solve tasks, raising questions about whether similar precautions were taken in this study.

Author response:
Yes. We take some measures such as recording the annotating process to prevent the use of LLMs.

Reviewer comment (Evaluation):
It is unclear why the paper studies the performance of the human evaluators on a per-evaluator basis rather than taking the majority agreement of the evaluators per piece of information and then reporting the results in aggregate.

Author response:
Here we aim to compare the detection difficulty of LLM-generated misinformation & human-written misinformation. Thus, we need to control the human evaluators the same for both LLM-generated misinformation & human-written misinformation.

Reviewer comment (Evaluation):
It is unclear how the attack rates in Section 3 are calculated. It is questioned whether these are based on manual evaluations from the authors and more details are suggested, such as how the generated pieces of information were annotated and how many people annotated each piece.

Author response:
Yes. We manually verify them. We will make it more clear in the revision.

Reviewer comment (Experiments):
The dataset seems to be not very large. I understand that for evaluating human detection difficulty, we can not use too large dataset. But the authors can enlarge the dataset for evaluation of machine learning model.

Author response:
We conduct a much more extensive investigation on misinformation generated by ChatGPT, Llama-7b-chat-generated misinformation (or 13b, 70b) and Vicuna-7b-generated misinformation (or 13b, 33b) to further validate our findings. For each LLM as the misinformation generator, we compare the detection difficulty of different misinformation generation methods including Paraphrase Generation, Rewriting Generation, Open-ended Generation and human-written misinformation. We include multiple human-written misinformation datasets including Politifact, Gossipcop and CoAID. The results are in Appendix A.

Reviewer comment (Experiments):
For detector difficulty, the authors only discussed the zero-shot detection of generative LLMs. The results on other kinds of models (i.e. in-context-learning boosted LLMs, soft-prompt based LLMs, and encoder-based Large models like BERT and its variants) are not discussed.

Author response:
We would like to explain why we adopt the zero-shot LLM detectors (e.g., GPT-4) to evaluate the detection difficulty of LLM-generated misinformation: (1) **The zero-shot setting can better reflect the real-world scenarios of detecting LLM-generated misinformation.** Conventional supervisedly trained detectors  (such as BERT in the pre-LLM era) are likely to be impractical to detect LLM-generated misinformation in the practices for the following two reasons. First, it is difficult to obtain factuality supervision labels to train detectors for LLM-generated misinformation since it is harder for humans to detect than human-written misinformation (Section 5). Second, malicious users can easily utilize methods shown in Table 1 and close-sourced LLMs (e.g., ChatGPT) or open-sourced LLMs (e.g., Llama2) to generate misinformation at scale in different domains, types, and errors, which is hard for conventional supervisedly trained detectors to maintain effective. (2) **Zero-shot Detectors can better reflect the intrinsic detection difficulty of  LLM-generated misinformation compared with human-written misinformation.** Conventional supervisedly trained detectors (such as BERT in the pre-LLM era) are strongly influenced by the labeled training set construction and cannot reflect the detection difficulty of test set well. For example, various factors in  the labeled training set construction (including adopting human-written or LLM-generated misinformation or their combination as the training set, mixing the misinformation data with different factual information data, the domains of training set, the percentage of misinformation data in the training set) can lead to completely different detection performances of human-written or LLM-generated misinformation test set. Zero-shot Detectors, which do not need labeled training set, can better reflect the intrinsic detection difficulty of human-written or LLM-generated misinformation test set. (3) **Zero-shot LLM detectors (e.g., GPT-4) are already almost the state-of-the-art misinformation detectors.** There are already many works that have shown directly prompting LLMs such as GPT-4 in a zero-shot way can outperform conventional supervisedly trained models such as BERT on detecting human-written misinformation [1,2,3,4,5]  

Thus, we adopt ChatGPT-3.5, GPT-4 and Llama2 with standard prompting and Chain-of-thought Prompting strategies as the **representative detectors** to evaluate the detection difficulty of LLM-generated misinformation data and also compare it with human-written misinformation data.

[1] A Multitask, Multi-lingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. https://arxiv.org/abs/2302.04023
 
[2] Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4 https://arxiv.org/abs/2305.14928
 
[3] Interpretable Unified Language Checking https://arxiv.org/abs/2304.03728 

[4] Fact-Checking Complex Claims with Program-Guided Reasoning ACL 2023 https://aclanthology.org/2023.acl-long.386/
 
[5] Assessing the Effectiveness of GPT-3 in Detecting False Political Statements: A Case Study on the LIAR Dataset https://arxiv.org/abs/2306.08190

Reviewer comment (Reproducibility):
Dataset is not opensourced. Actually, the data can be opensourced anonymously on GitHub.

Author response:
We release our whole datasets and detection results for reproduction and verification. https://anonymous.4open.science/r/llm-misinformation-iclr2024-7E35/

Reviewer comment (Reproducibility):
Will the dataset be opensourced once the paper is accepted?

Author response:
We release our whole datasets and detection results for reproduction and verification. https://anonymous.4open.science/r/llm-misinformation-iclr2024-7E35/

Reviewer comment (Experiments):
The study is relatively small number of evaluators and only evaluates a limited number of LLM-generated news items. This means that the findings of the study may not be generalizable to all LLM-generated news items.

Author response:
We conduct a much more extensive investigation on misinformation generated by ChatGPT, Llama-7b-chat-generated misinformation (or 13b, 70b) and Vicuna-7b-generated misinformation (or 13b, 33b) to further validate our findings. For each LLM as the misinformation generator, we compare the detection difficulty of different misinformation generation methods including Paraphrase Generation, Rewriting Generation, Open-ended Generation and human-written misinformation. We include multiple human-written misinformation datasets including Politifact, Gossipcop and CoAID. The results are in Appendix A.

Reviewer comment (Evaluation):
The study does not evaluate the effectiveness of different detection methods for LLM-generated misinformation. This means that it is not clear how well existing detection methods would perform at detecting the LLM-generated news items used in the study.

Author response:
We would like to explain why we adopt the zero-shot LLM detectors (e.g., GPT-4) to evaluate the detection difficulty of LLM-generated misinformation: (1) **The zero-shot setting can better reflect the real-world scenarios of detecting LLM-generated misinformation.** Conventional supervisedly trained detectors  (such as BERT in the pre-LLM era) are likely to be impractical to detect LLM-generated misinformation in the practices for the following two reasons. First, it is difficult to obtain factuality supervision labels to train detectors for LLM-generated misinformation since it is harder for humans to detect than human-written misinformation (Section 5). Second, malicious users can easily utilize methods shown in Table 1 and close-sourced LLMs (e.g., ChatGPT) or open-sourced LLMs (e.g., Llama2) to generate misinformation at scale in different domains, types, and errors, which is hard for conventional supervisedly trained detectors to maintain effective. (2) **Zero-shot Detectors can better reflect the intrinsic detection difficulty of  LLM-generated misinformation compared with human-written misinformation.** Conventional supervisedly trained detectors (such as BERT in the pre-LLM era) are strongly influenced by the labeled training set construction and cannot reflect the detection difficulty of test set well. For example, various factors in  the labeled training set construction (including adopting human-written or LLM-generated misinformation or their combination as the training set, mixing the misinformation data with different factual information data, the domains of training set, the percentage of misinformation data in the training set) can lead to completely different detection performances of human-written or LLM-generated misinformation test set. Zero-shot Detectors, which do not need labeled training set, can better reflect the intrinsic detection difficulty of human-written or LLM-generated misinformation test set. (3) **Zero-shot LLM detectors (e.g., GPT-4) are already almost the state-of-the-art misinformation detectors.** There are already many works that have shown directly prompting LLMs such as GPT-4 in a zero-shot way can outperform conventional supervisedly trained models such as BERT on detecting human-written misinformation [1,2,3,4,5]. 

Thus, we adopt ChatGPT-3.5, GPT-4 and Llama2 with standard prompting and Chain-of-thought Prompting strategies as the **representative detectors** to evaluate the detection difficulty of LLM-generated misinformation data and also compare it with human-written misinformation data.

[1] A Multitask, Multi-lingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. https://arxiv.org/abs/2302.04023
  
[2] Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4 https://arxiv.org/abs/2305.14928
  
[3] Interpretable Unified Language Checking https://arxiv.org/abs/2304.03728  

[4] Fact-Checking Complex Claims with Program-Guided Reasoning ACL 2023 https://aclanthology.org/2023.acl-long.386/
  
[5] Assessing the Effectiveness of GPT-3 in Detecting False Political Statements: A Case Study on the LIAR Dataset https://arxiv.org/abs/2306.08190

Reviewer comment (Experiments):
The study is relatively small and only evaluates a limited number of LLM-generated news items. This means that the findings of the study may not be generalizable to all LLM-generated news items.

Author response:
We conduct a much more extensive investigation on misinformation generated by ChatGPT, Llama-7b-chat-generated misinformation (or 13b, 70b) and Vicuna-7b-generated misinformation (or 13b, 33b) to further validate our findings. For each LLM as the misinformation generator, we compare the detection difficulty of different misinformation generation methods including Paraphrase Generation, Rewriting Generation, Open-ended Generation and human-written misinformation. We include multiple human-written misinformation datasets including Politifact, Gossipcop and CoAID. The results are in Appendix A.