CANDY : Benchmarking LLM s’ Limitations and Assistive Potential in C hinese Misinformation Fact-Checking (2025)

Abstract: The effectiveness of large language models (LLMs) to fact-check misinformation remains uncertain, despite their growing use. To this end, we present CANDY, a benchmark designed to systematically evaluate the capabilities and limitations of LLMs in fact-checking Chinese misinformation. Specifically, we curate a carefully annotated dataset of ~20k instances. Our analysis shows that current LLMs exhibit limitations in generating accurate fact-checking conclusions, even when enhanced with chain-of-thought reasoning and few-shot prompting. To understand these limitations, we develop a taxonomy to categorize flawed LLM-generated explanations for their conclusions and identify factual fabrication as the most common failure mode. Although LLMs alone are unreliable for fact-checking, our findings indicate their considerable potential to augment human performance when deployed as assistive tools in scenarios. Our dataset and code can be accessed at https://github.com/SCUNLP/CANDY .

## introduction misinformation, defined as "false or misleading information masquerading as legitimate news, regardless of intent" @xcite . with the largest internet user base, china produces vast amounts of such misinformation daily, making manual verification both labor-intensive and increasingly infeasible @xcite @xcite . large language models (llms) possess extensive knowledge and explanatory capabilities, positioning them as promising tools for misinformation detection @xcite @xcite . however, concerns about their effectiveness persist, such as hallucination limitations @xcite to this end, we introduce candy, a benchmark designed to systematically evaluate the capabilities, limitations, and practical roles of llms in factchecking chinese misinformation. to achieve this, we construct candyset, a curated multi-domain dataset consisting of approximately 20k news instances collected from mainstream chinese rumor refutation websites, with detailed sources provided in with candyset, we further expand our benchmark analysis using three tasks including fact-checking conclusion (i.e., if a statement is factual), fact-checking explanation (if a llm-generated explanation for its fact-checking conclusion is reliable), and llm-assisted fact-checking (to what extent can llms assist humans in fact-checking). to better identify and expose the deficiency of llms, we develop a taxonomy that categorizes flawed llm-generated explanations into three dimensions, which are further divided into seven finegrained categories (cf. section 3.2). using this taxonomy, we manually annotate around 5k llmgenerated explanations to analyze explanation deficiencies, contributing a valuable component of our dataset (cf. ## related works in recent years, with the proliferation of misinformation on social media, several works have developed various fact-checking benchmarks as shown in fact-checking benchmark language basic dataset information annotation for llm explanation human study claims time multi-domain ground-truth evidence explanation annotations explanation taxonomy various education level human-llm interaction records politifact(kao and yen, 2024) english 33721 ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ flawcheck(hsu et al., 2024) english 50 2019-2021 ✗ ✗ ✓ ✗ ✗ ✗ ✗ weibo(jin et al., 2017) chinese 9528 may 2012-jan 2016 ✓ ✗ ✗ ✗ ✗ ✗ covid19(yang et al., 2022) chinese 623837 jan-may 2020 ✗ ✓ ✗ ✗ ✗ ✗ ✗ checked(yang et al., 2021) chinese 2104 dec 2019-aug 2020 ✗ ✗ ✗ ✗ ✗ ✗ crossfake(du et al., 2021) chinese 219 feb-dec 2020 ✗ ✗ ✗ ✗ ✗ ✗ weibo21(nan et al., 2021) chinese 9128 dec 2014-mar 2021 ✓ ✗ ✗ ✗ ✗ ✗ mcfend(li et al., 2024b) chinese 23974 mar 2015-mar 2023 ✓ ✗ ✗ ✗ ✗ ✗ ltcr(ma et al., 2023) chinese 2290 dec 2019-jan 2023 ✗ ✗ ✗ ✗ ✗ ✗ chef(hu et al., 2022) chinese 10000 sep 2017-dec 2021 ✓ ✓ ✗ ✗ ✗ ✗ candyset(ours) chinese 20435 mar 2017-oct 2024 ✓ ✓ ✓ ✓ ✓ ✓ weibo21 @xcite and mcfend @xcite for multi-domain scenarios. however, except for chef @xcite , these benchmarks lack detailed explanations of supporting evidence, limiting their usefulness for deeper analysis. additionally, of all the benchmarks, only politifact(kao and yen, 2024) and flawcheck @xcite strive to get llms to generate explanations during the fact-checking process. however, both of these benchmarks fall short in providing detailed human annotations and thorough analysis of the accuracy or shortcomings of these explanations. moreover, none of the benchmarks consider testing llms in collaboration with real users in practical scenarios, which hinders their real-world applicability. our work addresses these limitations by introducing (1) carefully annotated evidence explanations for each instance, (2) constructing a taxonomy to categorize errors in llm-generated explanations, and ## task 1: fact-checking conclusion this section aims to evaluate the ability of llms to verify facts by assessing their performance in distinguishing factual statements from falsehoods. following @xcite , we adopt accuracy (acc.) and f1 score as evaluation metrics.